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(57) Abstract 

The system includes a panoramic input device (6, 7, 8), a computer 
processing system (15-18), and panoramic audio- visual presentation 
device (11, 12). The panoramic input device may include a sensor 
assembly (36) including a plurality of positionable radar (29, 38), camera 
(28, 37), and acoustical sensors (30, 39) for recording signatures from 
all sides of a subject simultaneously. The computer processing system 
processes signals from the input device to generate, update and display 
a virtual model. An interactive input device (10) is provided for 
allowing interaction with the virtual model. The panoramic audio-visual 
presentation device may be either a head-mounted display (1 1) or a closed 
structure (42) having display units mounted in all viewable directions 
surrounding a participant. The display units may be conventional (31), 
stereoscopic (32), autostereograpbic (33)> or holographic (34) display 
systems. 



PANORAMIC 
AUDIO SI&Ml 
PROCESSING* 



PANORAMIC 
VISUAL S/GWM. 
PROCESSING* 



n 




FOR THE PURPOSES OF INFORMATION ONLY 



Codes used to identify States party to the PCT on the front pages of pamphlets publishing international 
applications under the PCT. 



AT 


Austria 


GB 


United Kingdom 


MR 


Mauritania 


AU 


Australia 


GE 


Gcorgu 


MW 


Malawi 


BB 


Barbados 


GN 


Guinea . 


NE 


Niger 


BE 


Belgium 


GR 


Greece 


NL 


Netherlands 


BF 


Burkina Faso 


ffU 


Hungary 


NO 


Norway 


BG 


Bulgaria 


IE 


Ireland 


NZ 


New Zealand 


BJ 


Benin 


IT 


Italy 


PL 


Poland 


BR 


Brazil 


JP 


Japan 


PT 


Portugal 


BY 


Belarus 


K£ 


Kenya 


RO 


Romania 


CA 




KG 


Kyrgystan 


RU 


Russian Federation 


CF 


Central African Republic 


KP 


Democratic People's Republic 


SD 


Sudan 


CG 


Congo 




of Korea 


SE 


Sweden 


CH 


Switzerland 


KLR 


Republic of Korea 


SI 


Slovenia 


ci 


Cote d'lwire 


KZ 


Kazakhstan 


SK 


Slovakia 


CM 


Camerooo 


U 


. Liechtenstein 


SN 


Senegal 


CN 


China 


LK 


Sri Lanka 


TO 


Qud 


cs 


Gecboalovakia 


LU 


Luxembourg 


TG 


Togo 


cz 


Czech Republic 


LV 


Latvia 


TJ 


Tajikistan 


DE 


Germany 


MC 


Monaco 


TT 


Trinidad and Tobago 


DK 


Deumirk 


MB 


Republic or Moldova 


UA 


Ukraine 


ES 


Spain 


MG 


Madagascar * 


US 


United Stales of America 


fi 


Finland 


ML 


Mali 


uz 


Uzbekistan 


FR 


France 


MN 


Mongolia 


VN 


Viet Nam 


GA 


Gabon 









WO 94/16406 



PCT/US94/00289 



IMPROVED PANORAMIC IMAGE BASED VIRTUAL REALITY/ 
TELEPRESENCE AUDIO-VISUAL SYSTEM AND METHOD 

BACKGROUND OF THE INVENTION 

1. Field of the Invention 

This invention relates generally to panoramic display 
methods and more particularly to the sensor fusion of data 
from the panoramic arrangement of three-dimensional imaging 
sensors and surface contour sensors to form virtual objects 
and scenes, the processing of the virtual objects and scenes 
based on a viewer operating interactive computer input devices 
to effect the manipulation of the virtual objects and scenes 
defined in the computer, and the display of the effected 
virtual objects and scenes on a panoramic display unit to the 
extent that the viewer percevies that the virtual objects and 
scenes completely surround the viewer. 

2. Description of the Related Art 

My previous U.S. Pat. No. 5,130,794 describes a panoramic 
image based virtual reality system that incorporates a 
multi-lens camera system with spherical f ield-of -view (FOV) 
coverage. As shown in FIG. 2, objective lenses of the '794 
camera system face outward with adjacent or overlapping FOV 
coverage. The imagery from the camera is surface mapped onto 
the interior, of a three-dimensional ( 3-D ) shape defined in a 
special effects processor of a computer. Alternatively, the 
input source is at least one computer graphics system that 
generates three-dimensional graphics of spherical FOV 

1 
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coverage. The viewer operates interactive input device:: 
associated with the computer to manipulate the texture mapped 
virtual images. The virtual environment is instantaneously 
affected before the viewer and displayed on either a 
head-mounted display assembly or on contiguous display units 
positioned beneath, to the sides, and above the viewer. 

Limitations of the panoramic video camera system in '794 
are that the panoramic camera does not record a non- 
spherical field of viev(FOV) and does not incorporate a 
non-contact shape sensor. 

An improvement over the existing system is proposed in my 
Disclosure Document No. 197612, specifically Fig. 15, filed 
with the U.S. Patent and Tradmark Office in Feb. 1986, and in 
my recent paper entitled "Image Based Panoramic Virtual 
Reality System", presented at the SPIE/IS&T Symposium on 
Electronic Imaging: Science & Technology 92; Visualization, 
Holography, and Stereographies; Visual Data Interpretation, 
Paper No. 1168-02, on Feb. 9, 1992. 

In these documents a multi-lens camera system with 
positionable taking lenses is described. Taking lenses of the 
camera are faced inward or outward to record imagery of a 
subject in an continuous simultaneous manner. By combining 
panoramic visual field of view sensor data with associated 
shape sensor data a realistic panoramic image based 
three-dimensional computer generated model is rendered. 
Imagery from the camera is surface mapped onto the surface of 
a three-dimensional shape defined in a computer. The shape is 
input by a panoramic 3-D digitizer device. Audio data is 
input by a panoraimic 3-D audio system. Audio attributes are 
assigned to subjects in the model. Shape, imagery, and audio 
sensors may be combined to form one sensor array. Sensors are- 
positioned adjacent to one another to facilitate adjacent or 
overlapping coverage of a subject. Preferably corresponding 
panoramic shape, imagery, and audio signatures of a subject(s) 
are collected simultaneously. In this manner action of a 3-D 
subject is recorded from substantially all aspects at a single 
moment in time. The participant operates interactive input 
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devices associated with the computer to manipulate the virtual 
object. In one example, the participant observes the model on 
a head mounted display system. In another example, the 
participant is surrounded by contiguous audio-visual display 
units. In the latter example, each display unit displays a 
segment of the model. 

It is therefore the objective of this invention to provide a 
more versatile image based panoramic virtual reality and 
telepresence system and method. Still another objective is to 
produce systems and methods for recording, formatting, 
processing, displaying, and interacting with data representing 
3-D beings, objects, and scenes. More specifically, an 
objective of this invention is to provide a positionable 
multi-lens camera system for recording contiguous image 
segments of an object, being, adjacent surrounding scene, or 
any combination of these types of subjects; a signal 
processing means comprising first computerized fusion 
processing system for integrating the positional camera system 
with corresponding digitized shape and contour data; a second 
computerized fusion processing system for integrating first 
fused data with other fused data representing adjacent 
portions of a being, object, or scene comprising a panoramic 
computer generated model; where various 3-D digitizer systems 
may be incorporated for entering 3-D shape and contour data 
into a image processing computer; a third processing means to 
manipulate the geometry of subjects comprising the virtual 
model; a forth processing means for sampling out given fields 

of regard of the virtual model for presentation and 
distribution to display units and audio speakers; where signal 
processing means includes an expert system for determining the 
actions of subjects of the computer generated model; where the 
signal processing means includes image segment circuit means 
for distributing, processing, and display of the model; where 
the system includes a 3-D graphics computer system for the 
generation, alteration, and display images; and a system and 
method for image based recording of 3-D data which may be 
processed for display on various 3-D display systems to 
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include head n;ournecl display systems, and room display system;; 
with stereographic, autos tereosccpic, or holographic display 
systems. 

It is also an objective of this invention to provide 
interactive input devices operable by a viewer to cause the 
generation, alteration, display of 3-D images on said display 
assembly means; to provide associated 3-D audio systems; to 
provide alternative viewer interactive and feedback devices to 
operate the interactive input devices and associated 
processing means such that the resultant virtual environment 
is simultaneously effected before the viewers eyes; to provide 
an associated telecommunications system; and to provide a 
system for incorporation with a host vehicle, teleoperated 
vehicle, or robot. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 is a flowchart to which reference will be made in 
generally explaining the overall operation of the recording, 
processing, and audio-visual system 1 according to the present, 
invention. 

FIG. 2 is a perspective view of a cameraman carrying a 
panoramic camcorder system of spherical coverage described in 
prior art. 

FIG. 3 is a greatly enlarged fragmentary sectional veiw of 
one of the camera arrangements for optically recording image 
segments representing sides of a three-dimensional subject 
into a single frame according to the present invention. 

FIG. £ is a perspective view of a sensor array for recording 
accoustical, visual, and shape data for input according to the 
present invention . 

FIG. j is a side sectional view of the sensor array shown in 
FIG. 4. 

FIG. £ is a diagrammatic representation of an inward looking 
three-dimensional input source incorporating the sensor array 
shown in FIGS. ± and 5 . . 

FIG. 7 is a diagrammatic representation of a inward and 



WO 94/16406 



PCT/US94/00289 



outward looking panoramic: three-dimensional input source 
assembly incorporating the sensor array shown in FIGS. £ and 
7. 

FIGS. 8A-8D are d iagramat i c representations of video frames 
of three-dimensional coverage of beings and objects to be 
modeled in 3-D in the present invention. 

FIGS. 9A-9B are diagramatic representations of video frames 
of three dimensional coverage of beings, objects, and 
background scenes, respectively, to be modeled in 3-E> in the 
present invention • 

FIG. 10 is a diagramatic representation of a HDTV frame on 
which includes both foreground and background imagery 
necessary to model a virtual environment. 

FIG. U is a fragmentary view onto the top of the virtual 
world model in which recorded three-dimensional beings, 
objects, and/or scenes are incorporated according to the 
present invention . 

FIG. 12. is a fragmentary view onto the side of the virtual 
model shown in FIG. 11 . 

FIG. 13. is a diagramatic illustration showing how imagery is 
texture mapped onto a three-dimensional wireframe model to 
form a three-dimensional virtual model generated and processed 
for presentation by audio and video computer signal processing 
means of system 1.. 

FIG. 14_ is a perspective, partially diagramatic view showing 
an image based virtual model generated for audio-visual 
presentation by the visual signal processing means of system 
1 . 

FIG. 15 is a block diagram of an image formatting system for 
recording, processing, and display of an image of three- 
dimensional coverage which embodies the present invention. 

FIG. 16. is a block diagram of a second embodiment of system. 

FIG. 12 is a block diagram of a third embodiment of the 
system. 

FIG. 18. is a block diagram i llustrat ing /the incor por tat i on 
of a three-dimensional display system according to the present 
invention. *' 
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FIG. 19 is a. perspective, partially diagramatic view 
illustrating the three-dimensional viewing system described in 
FIG. 18. 

FIG. 20 is a block diagram of an embodiment of the present 
invention including a telecommunications system. 

FIG. 21 is a perspective, part ially diagramatic view 
illustrating the telecommunications embodiment according to 
FIG. 20. 

FIG. 22. is block diagram illustrating an embodiment of the 
present invention wherein a host vehicle control system with a 
panoramic sensor, processing, and display system provides 
telepresence to a viewer/operator for control of the host 
vehicle. 

FIG. H is a sectional view of a host vehicle incorporating 
the present invention shown in FIG. 22.. 

FIG. 24 is a block diagram illustrating an embodiment of the 
present invention wherein a remote control system for a 
remotely piloted vehicle with a panoramic sensor system 
transmits a three-dimensional panoramic scene to a control 
station for processing and spherical coverage viewing in order 
to assist the controller in piloting the te leoper ated veh icle . 

FIG. 25 is a perspective, partially diagrammatic view 
illustrating the remote control three-dimensional viewing 
system described in FIG. 2±. 

LISTED PARTS IN DRAWINGS 

1. : Improved panoramic image based virtual 

reality/telepresenc audio-visual system and method 

2. : Panoramic 3-D input source means 

3. : Panoramic 3-D signal processing means 

i: Panoramic audio-visual presentation means 
5: Suitable electrical interface means and assoiciated 
signal (general ) 

( 5a : video signal means) 

( 5b : digital shape signal means) 

( 5c : audio signal means) 
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F'anozamic 3-D camera system 
1; Panoammic 3-D digitizing system 
B: Panoramic 3-D audio recording system 
9.: Host computer system 
10 : Interactive input system 
11 : Head-mounted display(HMD) system 
12 : Large display assembly 
3 3 : Subject 

( 13a : being) 
(13b : object) 
( 13c : scene ) 

( -a : side a) 
(-b : side b) 
( -c : side c ) , etc . 
1 4 : Computer generated virtual world model 
( 14a : visual and shape model) 
(14b : audio model) 

( -a : modeled being) 
( -b : modeled object) 
f -c : modeled scene) 
1 5 : First processing means; fusion processor to wed shape 

and image segments . 
16 : Second processing means; fusion of model segments. 
17 : Third processing means; host simulation computer for 

manipulating world model; geometry processor. 
18 : Forth processing means; image processing for display and 

distribution. 
19 : Computer graphics system 
20 : VRT telecommunications system 
21 : VRT vehicle control system 
22 : Artificial intelligence system 
23 : Audio processing system 
24 : Participant; viewer/operator) 
( 24a : first participant) 
(24b : second participant) 
25 : Mass storage device 

( 2 5a : visual and shape mass storage) 
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( 2 5 b : audio data mass storage) 
2_6: Panoramic model segments 

( 26a : , visual and shape model segment) 

( 26b : audio model segment) 
27 : Sensor ( s ) 
28 : Image sensor (s) 
29 : Shape sensor (s) 
30 : Audio( Accoust ical ) sensor(s) 
31 : Conventional display unit(s) 
3 2 : Stereographic display unit(s) 
33.: Autostereoscopic display unit(s) 
3±: Holographic display unit(s) 
35: Audio speaker(s) 
36 : Sensor array 
37 : Camera 
38 : Radar 
39 : Microphone 
40 : Array housing 

41.: Overlapping field of regard coverage of sensors. 

42: Edge of adjacent field of regard coverage of sensors. 

43.: Rigid transparent support 

4±: Array assembly 

45 : Screw 

4_6: Support armature 

4_7: Panoramic optical assembly arrangement 
4S : Objective lens 

49. : Light sensitive surface of the camera 

50: Fiber-optic image conduit ( bundle ) 

51 : Focusing lens 

52 : Camera housing 

53 : Charge Coupled Device(CCD) 

54 : Sheathing of image conduit 

55: Shape data (wireframe ) representing subject model 
!5£: Image data representing a subject 
( 56a : being) 

(56b : object ) ' . 

( 56c : scene ) 
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57 : Audio data representing a subject 

( 57a : being) 

(57b : object) 

( 57c : scene) 
58 : Viewing space 
59 : Head position of participant 
60 : Hand location of participant 
61 : Sample frame of panoramic camera 

62: Transimitter ;. for transmitting an over-the-air stereo 
aud i o s ignal . 

6 3 : Receiver; for recieiving an over-the-air stereo audio 

s ignal . 

64 : Stereo audio headphones 

65 : Structural supports of the large display assembly 
66 : Graphics input system 
67 : Videotape player 
68 : Videodisc player 

69.: Video analog-to-digital converter 

70 : Display unit; generally; may include audio system. 

11: Displayed scene 

72 : Image segment circuit means 

73 : Image control unit (including chasis, processors, etc.); 

may include audio means. 
74 : Polygonal surfaces of model *14a 

7 5 : Head position of viewer 

76 : Position sensing system sensor 

Position sensing system source 
78 : Position sensing system electronics unit 
79 : Audio signal to means 4 
80 : Video signal to means 4 
81 : Display unit viewing surface 

82 : Position and orientation data and associated conductor 
from interactive input system 10 

8 3 : Source conductor line 
84 : Sensor conductor line 
8 5 : Radar antenna 

86 : Radar waveguide 

9 



WO 94/16406 



PCT/US94/00289 



£Li. : ^:ada: tr snsmi ttcr /r ec iever 
88. : Master clock 

S_S: Conventional signal router/switcher 

3-D display unit; generally. 
£1: 3-D display system embodiment of system 1. 
S2: Encoder/compressor 
93 : Encryptor 
94 : Modem . 
95 : Decryptor 
9_£: Decoder/expander 

9J7; Non-contact position and orientation sensor system (i.e. 
Radar or LADAR); may include camera system. 

98. : Digital data network 

99. : Telephone line 

100 : Edge of projected image 
101 : Floor of large assembly 
102 : Host vehicle 
10_3: Host vehicle controls 

10±i Host vehicle control surfaces and motors. 

105 : Rear projection screen 

IP 6 : Entry/exit assemblies for assembly 12 

107: Structural support, framework, and fasteners for large 

assembly 12. 
108 : Remotely piloted vehicle 

109.: Tranceiver; for sending and recieving radio 

f reguency(RF) over-the-air digital data. 
iiCt: Over-the-air RF digital data link 
HI* Participant support means 
.1 1 2 : Remote vehicle control system 
113/- Remote vehicle control surfaces and motors 
114 : Remote vehicle manipulators 
115.; Timing signal conductor 
116: Model signal conductor 
112: Processing means for conventional TV 
118: Processing means for stereo display TV 
119: Processing means for autostereoscopic TV 
2_20: Processing means for holographic tfv 
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221 : Processing and distribution system for image segment 
circuit means 

222 : Audio-visual units of image segment circuit means 
2 23 : Hemispherical scan of LADAR system; may include 

integral registered camera system. 
224 : Near field of view of LADAR system; may include 

integral registered camera system. 
225 : VRT control station for remotely piloted vehicle 
226 : Video compression and data system (including 

communications buffer ) 
227 : Video decompression and data system (including 

communications buffer ) 
228 : Peripherial devices 

DETAILED DESCRIPTION OF THE 
PREFERRED EMBODIMENTS 

As required, detailed embodiments of the present invention 
are disclosed herein; however, it is to be understood that the 
disclosed embodiments are merely exemplary of the invention 
which may be embodied in various forms. Therefore, specific 
structural and functional details disclosed herein are not to 
be interpreted as limiting, but merely as a basis for teaching 
one skilled in the art to variously employ the present 
invention in virtually any appropriately detailed structure. 

For clarity of description, a preliminary summary of the 
major features of the recording, processing, and display 
portions of a preferred embodiment of the system is now 
provided, after which individual portions of the system will 
be described in detail. 

Referring to the drawings in more detail. 

As shown in FIGS . 1 the reference 1 generally designates a 
system and method of rendering and interacting with a 
three-dimensional (3-D) computer generated model that 
comprises the virtual real ity/telepresenc'e system 1 presented 
to a participant 2±. The system 1 generally includes a 
panoramic input source means 2, panoramic signal processing 
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means 3, and panoramic audio-visual presentation assembly 
means 4. connected generally by suitable electrical interface 
means 5. Electrical interface means 5 7 including the 
appropriate conductor, input/output port or jack 
interconnections, and associated signal, is indicated by lines 
and arrows in the drawings. Optional or alternative interface 
means and devices are indicated by dashed lines in the 
drawings. Input means 2, generally consists of a panoramic 
3-D camera system 6, a panoramic 3-D digitizing system .7, and 
a panoramic 3-D audio recording system Input means 6_, 1_, 

and 8. include a plurality of respective sensors that are 
positioned to record geographic and geometric subjects. A 
subject 13. may comprise three-dimensional beings or things in 
the real world. The world model 14. comprises a model 14a that 
includes shape and imagery, and an audio model 14b that 
includes accoustical recordings. The audio model corresponds 
to the shape and imagery model. Preferrably, all sides of 
the subject are recorded simultaneously by. the input means £, 
7, and £. 

Signal processing means 3. preferrably includes a first 
computer processing means 15> for sensor fusion of the 
resulting imagery signal 5a. and shape data signal 5b. The 
first processing means operates on the signals 5a_ and 5b. to 
combine shape and surface data of corresponding segments of 
the 3-D subject. The resulting 3-D model segments 26a are 
portions of the computer generated world model 14a . Signal 

processing means 3- preferrably also includes a second computer 
processing means 1£ for fusion of imaging and shape data 
segments 26a derived by first apparatus 15. to form a 
continuous panoramic world model 14.. Signal processing means 
2 also includes a third computer processing means 12 for 
manipulating the computer generated model 14a . The third 
processing means is typically operated to perform interactive 
3-D visual simulation and teleoperated applications. Signal 
processing means 3. also includes a forth computer processing 
means 18. to sample out and transmit image scene 2i segments of 
the world model 14a to each respective display unit of the 
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audio-visual assembly means ± . Means 2 includes includes 
processing means for interface with input sources 2, 
peripherial computer data entry and manipulation apparatus 
refered to as an interactive input system 1(3, and assembly 4_. 
Signal processing means 15., 16., JJ7., UL, and 2_3 include a 
central processing unit, terminal bus, communication ports, 
memory, and the like typical to a conventional computer(s). 
Operating system software, board level software, processing 
data, generated images and the like are stored in mass storage 
devices 25. which may include disk drives, optical disk drives, 
and so forth. All signal processing means 15., 16_, 3/7, UL and 
23 may be incorportated into a single computer 9 or a 
plurality of networked computers j[_9 tfo the nth) housed in a 
single or separate chassis. Additionally, means 3 may include 
a computer graphics system 19_, a telecommunications system 20 , 
a vehicle control system 21_, or artificial intelligence system 
22 to perform special processing functions. Special 
processing systems 19., 2^0, 2_1, and 22_ may be integral or 
networked to computer 

Audio sensors 30. are faced inward about a subject or outward 
to record signals representing audio segments 26b of a 
surrounding subject. Preferrably, the image, shape, and audio 
sensors 2JL, 29_, and 30. respectively, are positioned adjacent 
to one another and record a continuous corresponding subject 
13 . The audio processing system 21 receives recorded audio 
signals 5c from the panoramic 3-D audio input system 13. The 
audio signals 5c_ as assigned to modeled subject 14a comprise 
an accoustical world model 14b . The audio model 14b is 
continuously updated by the computer 2_3 based on data recieved 
from the interactive input system iO.. Computer 9. communicates 
changes to the world model 1± via digital data link 
interconnected to computer 2_3. Audio means 2_3 includes * 
processing means and software means for the generation of 3-D 
audio output in response to changes and actions of subjects 
modeled in the computer generated model 1 4a . The output audio 
signals are transmitted to speakers positioned about the 
participant by way of the panoramic aadio-visual assembly 
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means 4_. 

The preferred embodiment of the system 1 may generally 
comprise two alternative panoramic audio-visual assembly mean 
1: A heartmounted display (HMD) assembly 11_, or a large 
display assembly 12. The large display assembly 12. may 
incorporate conventional 3_1 7 stereograph ic . 22 , 
auto-stereographic 3.3, or holographic 3± display units. 
Specific processing means IB. compatable with a given display 
unit's 31., 32., 33./ or 34. format operate on the virtual model 
l_4a_. The processing means 18. then outputs a signal 
representing a model segment 26a to a predetermined display 
unit 31, 32, 31 or 3±- Display units 31, 32, 33., or 34. are 
placed contiguous to one another in a communicating 
relationship to the participant such that a continuous scene 
is presented to the participant. In this manner the same 
basic imagery, shape, and audio data is rendered into a model 
14a. that may be operated upon for presentation on 
conventional, stereographic, autos ter eoscopic, or holographic 
display systems. 

The model 14. presented to the participant may be derived 
from prerecorded data stored in a mass storage device 25. 
Alternatively, a live feeds from input sources 2 at a remote 
location are processed in near real time and the participant 
can interact with the remote location by using teleoperated 
devices. In these manners the viewer is immersed in a highly- 
interactive and realistic computer simulation. 

Still referring to FIG. 3~, in operation a panoramic sensor 
array comprising a plurality of shape, visual, and aural 
sensors are positioned to record a three-dimensional subject 
in a substantially continuous panoramic fashion. Each sensor 
22 outputs a corressponding signal specific to that sensors 
field of coverage. Signals representing visual and shape data 
are transmitted from input sources 6 and 7 to the signal 
processing means 2* A first computer processing means 15 
fuses the shape and visual signals to form^ model segments 26a . 
The pool of model segments are then transmitted to a second 
processing means 16 that fuses or matcttes adjacent and 
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corr esond ing model segments to one another. The matching cf 
intersections of the pool of model segments yields a panoramic 
three-dimensional model 14a . Typically the model 14a is 
rendered such that three-dimensional subjects in the 
foreground are of high-resolution and three-dimensional 
subjects in the background of less resolution. Preferrably, 
the background scene lies approximately ten feet beyond the 
boundary of the furthest distance the participant would 
venture into the virtual model. This is because beyound ten 
feet perspective is not significantly perceptable to the 
average human. And beyound this viewing distance the 
background scene of the model 14a would not need to be 
rendered in a 3-D manner because the viewer can not perceive 
parrallax and hence the the realism is not increased. A third 
processing means 1_7 receives the fused model of panoramic: 
coverage. The third means manipulates the geometry of the 
model 14a based on viewer interaction. A forth processing 
means 18, samples out portions of the model and transmits 
signals representing scenes 71 of a given field of view to 
predetermined display units of the display assembly 11_ or 12 . 
The dimensions and detail of the virtual model 14_ may be 
increased by moving the sensors to different locations 
throughout the real world environment in order to increase the 
resolution of the recorded subjects and to increase the pool 
of perspective views of subjects throughout the recorded 
environment. These sensor recordings are then processed and 
added to the existing data base and existing model in the same 
manner as prior subjects modeled for inclusion in the computer 
generated environment. Simultaneous with visual input, 
processing, and display, audio sensors 3£ transmit audio 
signals to an audio processing system 23.. The audio 
processing system is operated to assign audio signals to 
visual subjects positioned and comprising the panoramic 
computer generated model. 

An interactive input system 10., such as ,a position sensing 
system, monitors the viewers head position. Position data is 
transmitted to the visual and audio simulation processing 



WO 94/16406 



PCT/US94/00289 



system 17 -and 23_ respectively. The position and orientation 
data from system 1£ is processed by the visual and audio 
simulation processing means to update the model 14_ after each 
of the participants actions. Updating the model typically 
involves the participant moving a virtual object in the model 
with his hand, or changing the viewpoint of the displayed 
scene based upon a change in the participants head position. 
Positional changes of objects, subjects, and scenes are 
continuously updated and stored in the memory of the computer 
9 . 

Imagery and audio signals are transmitted from the visual 
15-18 and audio 2_3 processing means to the audio-visual 
assembly means 11 or l^.. The processing means has appropriate 
output processors, conductors, and interface connections to 
transmit the visual and audio signals to the visual display 
units 3i, 32, 33_, or 34. and audio speakers 35, of the display 
assemblies 1JL or 12. The visual model 14a and aural model 14b 
are updated and displayed instantaneously before the viewers 
eyes. 

INPUT MEANS 

Referring to FIG, 1 in more detail, input means comprises a 
3-D camera system 6j_ 3-D digitizing system 7_, , and 3-D audio 
system 8. Pr ef errably , at least one image sensor 18 of each 
image system, at least one shape sensor 29_ of each 3-D 

digitizing system, and at least one accoustical sensor of 
at least one audio system are positioned adjacent to one 
another and record a continuous corresponding segment of the 
subject 12. FIG. 2 illustrates a panoramic camera system 6 of 
prior art in which a plurality of image sensors 28a-28f and 
audio sensors (not shown) are faced outward about a point or 
area to record a contiguous surrounding visual subject scene 
13c,. FIG. J illustrates a panoramic camera system in which 
image sensors 28a-28f are positionable and' may be faced inward 
to record representations of each side of a subject 13 . 
FIG. i and 5 illustrates a sensor array 36. including a 
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visual system comprising a small conventional camera 27_/ a 3-D 
digitizing system comprising a small conventional radar 3S , 
and an accoustical system including a microphone 3J9. The 
microphone, radar, and camera of each array have overlapping 
field-of -regard coverage 41.. The overlapping coverage enables 
each the arrays sensors to record an accoustical, shape, and 
image signature of a given side of a subject 1_3_. FIG. 6. 
illustrates a plurality of arrays 36a-36f faced inward about a 
3-D subject. Each array has adjacent f ie ld-of -regard coverage 
42 of the subject such that each side of the 3-D subject is 
recorded. Accoustical, shape, and image signatures from each 
of the arrays are transmitted to signal processing means 3.. 

FIG. X illustrates sensor arrays may be faced both inward 
and outward to record a subject. Arrays are positioned 
adjacent to one another to form a panoramic array assembly 4 4 . 
Sensors of the adjacent arrays 36a-36 f of the assembly are 
positioned to have adjacent f ield-of -regard coverage 12.< 
array assembly has a substantially panoramic 3-D spherical 
f ield-of -regard coverage about a point. A plurality of array 
assemblies 4 4a-44 f may be arranged in the real world to 
simultaneously record a subject 13. environment from various 
points-of -regard . In this manner, virtually all sides of a 
subject surrounded by the array assemblies is recorded and 
backgroud scenes surrounding the subject are also 
simultaneously recorded. Alternatively, a single assembly 4_4 
may be moved thru space in the real world and records a 

subject environment from various points of regard at 
different times. The array 36. or array assembly 44, may be 
constructed in a portable fashion such that the array or array 
assembly are carried through a real world environment by a 
living being or vehicle. Each array of the assembly transmits 
its respective accoustic, shape, and imagery signatures to the 
processing means 3.* Processing means operates on the 
signature data to render the virtual world model 14.. Array 36. 
and array assembly 4_4 may be fastened together and supported 
by conventional means such as screws 4_5 .and support armature 
4G , Furthermore, sensors may be distributed over a vehicle 
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such that the inner or outer skin of the vehicle becomes- a 
housing for the sensors. The sensors can be placed on remote 
or host, piloted or unpiloted vehicles. 

1) THREE-DIMENSIONAL PANORAMIC SHAPE INPUT 
A panoramic 3-D digitizing system 7 comprises one type of 
input source 2 and is operated to input 3-D data representing 
a subject 13^. The system 7_ 1S operated to record the shape of 
a 3-D subject. System 7 may comprise a 3-D light pen, optical 
scanner, image recognition system, sonar, ultrasonic, laser 
scanner, radar, laser radar (LADAR) system or systems. 
Additionally, mathematical formula defining the shape of the 
subject may be entered by an operator via a keyboard. The 3-D 
data is transmitted from the system 1 to a computer processing 
system 9_ were it is operated upon. As shown in FIG. 1J3, the 
resulting 3-D data representing the subject is called a 
wireframe 5_5. The wireframe is a 3-D line and point computer 
generated rendering of a subject. The intersection of the 
lines form polygons that define the surfaces of the subject. 
A 3-D shape input system including a stylus and model table 
arrangement of the type described in U.S. Pat. 4,514,818 by 
Walker available from Quantel Limited, UK, or the 3SPACE TM 
Digitizer available from Polhemus of Colchester, VT may 
provide the shape data in system 1. Alternatively, a 
three-dimensional input system of a type described in U.S. 
Pat. 4,737,032 and 4,705,401 by Addleman and available from 
Cyberware Labratory, Inc. as the Rapid 3D Color Digitizer 
Model 3030 and associated products may provide the shape data 
in system 1. The Cyberware digitizer incorporates sensing and 
illumination elements to record a three-dimensional subjects 
shape and color. Seconds later, a graphics workstation 
displays the object as a detailed, full color, 
three-dimensional model. Alternatively, a radar and camera 
system decribed in U.S. Pat. 5,005,147 by Krishen et. al. may 
be incorporated to provide shape and imagery data in system 1_. 
Still alternatively, a laser-radar (LADAR ) system, including a 
video camera, available from Autonomous Technologies Corp. of 
Orlando, FL, may be incorporated to provide shape and imagery 
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data in the system I. 

2) THREE-DIMENSIONAL PANORAMIC CAMERA INPUT 

Preferrably, a 3-D camera system 6 comprises a plurality of 
objective lenses typically faced inward about a being or 
object, and outward to record a scene, Preferrably the 
objective lenses 4 8a-48f of the camera has overlapping or 
adjacent field of view . coverage . Any conventional TV camera 
or non-standard camera may be utilized in the present system 1 
that is compatable with signal processing means 2< The 
electrical section of the camera is structured to convert the 
visual images recieved by the image processor into electrical 
video signals 5a such that the information is in a format that 
is compatible with standard video processing equipment. Any 
conventional or unconventional video camera 37. may be adapted 
to accept the images from the disclosed optical systems in 
FIG. 1 thru FIG. 2- The image processor of the camera is 
structured to convert the visual images received into 
electrical video signals. Preferrably, the processed camera 
signals are typically standard synchronized coded signals 
utilized in the United States for video transmission. The 
signal processor 3. may be modified to convert each received 
electrical video signal 5a_ from the image processor means into 
a standard or non-standard synchronized coded signal of any 
given country or format for transmission and processing as 
desired, such as NTSC, PAL, SECAM, IDTV, HDTV, or the like. 

In both the spherical field of view optical assembly of FIG. 

2-, and the positionable field of view camera arrangment of 
FIG. 3./ images may be combined by either electronic means or 
by optical means. Similarly, image chrominance, luminance, 
hue, and intensity may be controlled electronically, 
optically, or electro-opt ically by the camera or later by the 
signal processing means. Typically, when a plurality of 
cameras 6a-6f are incorporated, the plurality of images are 
compressed into a single frame by processing means 2. When a 
single camera £ is incorporated, the images are optically 
integrated into a single frame. 

Any of these arrangements may be incorporated with array 35 , 
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or array assembly 4_4 of the system 3^. 

Although simple optical systems are depicted in FIGS. 2-7 , 
it should be clear to one skilled in the art that more complex 
optical arrangements can be employed. Other optical elements 
and electro-optical components taht may be included are 
automatic shutters, automatic focusing devices, optical 
filters and coatings, image intensif iers, correcting and 
inverting lenses, lens adapters, sensitive recording surfaces 
and medium of. various types and formats, wavelengths, and 
resolutions, and so forth and so on. These various optical 
arrangements are given the designer to accomplish a given, 
task. Standard video compression devices can be incorporated 
into the camera to compress the signal to aid in the storage, 
transmission, and processing of each image. Image sensors 
associated with moving target indicators (MTI), pattern 
recognition systems, and so forth, may be integrated with the 
optical systems of system !L. Conventional over-the-air video 
transmitters can be incorporated to transmit each image to a 
remote video receiver for processing and display. 

Pig. 2. illustrates a prior art camera used for recording a 
scene of spherical field of view coverage. A type of 
spherical camera utilized is generally of the type described 
in U.S. Patent 5,130,794 by the present inventor . As shown in 
FIG . 9A, in this way a spherical field of view scene about a 
point is recorded on a single frame. FIG. 3 illustrates a 
positionable 3-D camera system for recording all sides of 
subjects to be modeled in the computer generated virtual 
-environment. Images are transmitted from the objective 
lenses, thru fiber optic image conduits in focus to a light 
receiving surface 49_ of a camera 37_. As shown in FIGS. 8a_ 
thru 10., in this way all sides of a subject or subjects are 
recorded in single frame 6JL. Typically, optical elements, as 
shown in FIG. 3, are arranged to record images of various 
sides of the subject into a single frame. Optical elements 
such as mirrors, prisms, or coherent fiber-optic image 
conduits transmit images from one or more objective lenses to 
the light sensitve recording surface ©f the camera or cameras. 
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The optical fiber bundles are of a type that allows them to be 
twisted in route such that the desired image orientation at 
the exit end of the bundles is facilitated. Image bundlles 
have a rectangular cross section to facilitate the format of 
the image with the cameras format. Cross sectional resolution 
of 60 to 100 fibers per millimeter and low light loss 
transmission construction are incorporated to maintain image 
quality. Optical fibers are gathered at their exit end such 
that the image fibers optical axes are parallel to one another 
and perpendicular to the light sensitive recording surface of 
the camera. The image focused on the exit end of the conduit 
is optically transmitted in focus to the light sensitive 
surface of the camera. FIG. 9b is a frame in which a 
plurality of subjects whose sides have been recorded by a 
camera of a type generally illustrated in FIG . 2- 
Alternatively, FIG. 10. shows that imagery of all beings, 
objects, and background scenery comprising model 14a may be 
combined in a single frame. Preferrably, a high resolution 
sensor, such as a HDTV or IDTV recording system, is 
incorporated. High resolution recording allows for images to 
be enlarged for later viewing and still retain acceptable 
detail . 

Alternatively, as shown in FIG. 5, the panoramic camera 
system may comprise a plurality of cameras. When a. plurality 
of cameras are incorporated, images are recorded by 
positioning a plurality of image sensors toward and about a 
subject such that substantially continuous coverage is 

achieved by the sensors. Typically, conventional charge 
coupled devices (CCD) 53. positioned directly behind objective 
lenses or fiber optic image conduits to record image segments 
of beings, objects, and scenes. Alternatively, image conduits 
50 may be routed to a pluraity of cameras. The images are 
then transmitted to television production equipment, or 
computer 9_ over a conductor for processing. U.S. Pat. 
5,130,794 by the present inventor and U.S Pat. 5,023,725 by 
McCutcheon disclose optical and electronic methods and means 
of recording and compressing a plurality of images into a 
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single frame generally appllicable to incorporation in the 
present invention 1_. Alternatively, instead of spatially 
compressing or spaitially multiplexing a plurality of images 
into a single frame, images may also be time multiplexed. In 
such an arrangement, alternating images are electronically 
sampled into a single channel* The images are later, 
demultiplexed by a video demultiplexer device and processed by 
the computer 9.. A video multiplexer/demultiplexer system of a 
type generally incorporated into the present system 1 is 
available from Colorado Video Inc. as the Video Multiplexer 
496A/B and the Video Demultiplexer 497A/B. While digital 
compression, spatial compression, or spatial multiplexing the 
images is not required, it will be appreciated by those 
skilled in the art that compressing the image in one of these 
manners greatly assists in the transmission, processing, and 
storage of the vast amount of imagery necessary to build a 
panoramic model. 

3) THREE-DIMENSIONAL AUDIO INPUT 

The 3-D audio input system 8_ preferably is in communicating 
relationship with a high speed digital audio signal computer 
processing system 22. that delivers high quality three 
dimensional sound over conventional headphones. In Fig. 2 
microphones 39a-39f (not shown) are distributed to face 
outward from the lens housing to record a spherical 
accoustical field of regard coverage about a location. In 
Fig- 3, microphones 39a-39 f (not shown) are faced inward about 
a subject to record a contiguous accoustical field of regard 

coverage eminating from the subject 12- As illustrated in 
FIG. ± and 5, microphones may be integrated with the array 36 . 
The microphone 39. of each array preferrably have audio 
coverage 41_ corresponding to the shape and optical sensor 
coverage 41. of the same array. FIG. 6 and 7. show that arrays 
may be placed beside one another to achieve continuous 
adjacent panoramic audio coverage 42 of a subject. Audio 
signals 5c from audio input sources 6c-6f /are transmitted to 
the computer 23.. The computer 2_3 may consist of a . 
conventional personal computer or computer workstation 9_ and 
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includes printed circuit boards designed to sample the input 
audio signals 5c_. Types of workstations and associated 
printed circuit boards generally utilized for 3-D audio input, 
processing, and output are manufactured by Division Inc. of 
Redwood City, CA as the Accoustetron complete studio quality 
audio workstation and the Convolvotron and Beachtron audio 
processor printed circuit boards. The audio system 23 uses 
the virtual 3-D audio client/server protocol standard (VAP) as 
an operating system software interface with audio input 
devices such as compact disc or magnetic tape machines, and 
for communicating with means 17_ of computer 9_. The boards 
occupy ISA-compatable personnal computer slots. Input 
accoustical data from audio input source 8a-8f are typically 
stored on digital or analog sources such as compact disc or' 
magnetic tape machines, or may be digitized and stored in 
computer memory 25b and referenced to beings, objects, and 
scenes that form the visual model 14a comprising the computer 
generated environment 14 , 

The computer 2_3 samples and affects recorded audio data 
comprising the audio model 14b based on control data including 
audio sources position and orientation, participant position 
and orientation, environment reflection and transmission 
attributes, and audio source and participant velocities in the 
virtual environment 14. The host computer processing means 1J7 
is programmed with software such that control data is 
automatically transmitted over a standard RS-232C output of 

means to audio means 2^-. Control data is operated upon by 
computer 23 as a local geometric transform and HRTF tables and 
gains and input into a high spped digital audio signal 
processing system (e.g. Convolvotron or Beachtron) printed 
circuit boards. The input data is operated upon by the board 
level system of means 23, to affect input audio sources 
corresponding to each board level system. Input audio sources 
£ are converted from an analog to digital audio signal 5c. 
The board level system then output the affected audio to a 
digital to analog converter. The audio system 23_ outputs two 
independent 16-bit digital/analog converters synchronized to 
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the analog to digital converters driving conventional stereo 
output. The audio is typically transmitted over conductors to 
the stereo microphones on the participants 2_4 head. In this 
manner the perceived locations of sound sources in the 
environment can remain independent of the orientation of the 
user . 

Alternatively, audio computer 23_ may comprise a personal 
computer or workstation that access analog or digitally 
recorded sound data that is input or stored directly in 
computer 23. or. stored on disk, digital-audio tape, or compact 
disc. Computer 23. uses the Musical Instrument Digital 
Interface (MIDI) audio system to program, process, and output 
MIDI fomatted data streams which in turn are interpreted by 
MIDI, devices such as synthesizers, drum machines, and 
sequencers distributed accross the MIDI network. The system 
outputs a stereo signal to the participant 2_4 which typically 
wears stereo headphones 6J_. The audio system 23. may be of a 
type generally available from Silicon Graphics Inc., CA as the 
IRIS Indigo or Personal IRIS 4D/35 workstation, which include 
DAT-quality digital-audio subsystem, and configured with Audio 
Library software; and available from VPL Research Inc. CA as 
the AudioSphere TM system for use with computer generated 
virtual environments. Optionally, the MIDI audio system may 
be designed integral to computer 9_ (e.g. available on all 1993 
and future Silicon Graphics Inc. platforms). 

Alternatively, the stereo audio signals 5c can be 

transmitted over-the-air to a reciever 6-3- on the headphones 64 
by an infrared or radio frequency device. An over-the-air 
audio system of a type generally incorporated into the present 
system is available from Radio Shack Inc. as the Wireless FM 
Microphone with tranceiver and the Radio Shack Inc. Stereo FM 
Radio Headset. The transceiver 6_2 transmits an over-the-air 
stereo radio signals output by the computer system 2J3 tunable 
between 88-108 GHz to the reciever £3 of the an FM radio with 
stereo audio headphones 6_4 worn by the participant. 

Alternatively, the audio signals 5c from audio input sources 
can be transmitted by conductors direitly to speakers 3^ 
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distributed around the participant. The speakers are 
supported by the structure 6_5 of the large display assembly in 
a manner consistant with U.S. Pat. 5,130,794 by the present 
inventor or that described in U.S. Pat. 4,868,682 by Shimizu 
et al. 

4) GRAPHIC COMPUTER AS INPUT SOURCE 

Still alternatively a graphics computer 19. is operated as an 
input source 2_ (not shown) to create a 3-D world model 14a . 
The computer system includes a digital computer including a 
central processing unit, memory, communications ports, and the 
like. Operating system software, graphics software, 
processing data, generated images and the like are stored in 
mass storage devices which may include magnetic disk drives, 
optical disk drives, and so forth. Commands to operate the 
computer system and graphics generation commands are entered 
by means of the viewer interaction devices which may include a 
keyboard and graphics input device. The graphics input device 
66. may consist of one or more of a joystick, a trackball, a 
"mouse", a digitizer pad, a position sensing or pointing 
system, a non-contact viewer position and recognition sensing 
system, a voice recognition system, or other such devices. 
The graphic input device £6 may be operated as an input source 
2 or as part of the participants interactive input system 10.. 
The computer graphics system 19_ includes a bit mapped video 
display generator wherein each pixel element or pixel is 
accessible for generating high resolution images. The video 
display generator is connected to an input port by suitable 
conductor lines. The computer generated images are then 
further processed by the signal processing means 3. for 
display. The digital computer may be any type of computer 
system which has the required processing power and speed such 
as the type which are employed in 3-D computer graphic 
animation and paint applications. The computer system may 
function as a simulator controller if the display means of the 
present invention are used as simulators or as a game 
controller if the systems are employed as arcade games. The 
computer system may also be used to create special visual 
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effects by combining artificial and animated scenes with live 
camera recorded scenes. 

A graphics computer system of a type utilized herein is 
generally of the type, manufactured by USA Quantel Inc., 
Stamford, CT as the Quantel Graphics "Paintbox" TM System, or 
by Alias Research Inc., Toronto, Ontario, Canada as Alias 
PowerAnimatoij TM animation software for operation on Silicon 
Graphics Inc. workstations. It is forseen that the graphics 
computer system 19. and image processing system 11_ may occupy 
the same computer 9_. 

Additionally, conventional videotape 6J7 and videodisc 68_ 
player % s input signals 5c representing prerecorded image and 
audio signals may transmit images to the signal processing 
means 3. As illustrated in FIGS , 1_0 thru 12 each frame may 
consist of images of one, several, or all subjects to be 
modeled in the virtual environment. Likewise, a single, 
several, or all the audio tracks may be recorded onto a single 
recording medium. 

Additionally, a computer mass storage 25. database may serve 
as an input source. In such an instance, shape, imagery, and 
audio information may be encoded onto tape, or a magnetic or 
optical diskette or platter in an all digital format. 

PROCESSING MEANS 

Processing means 3 of system 1 at least includes a host 
computer system 9- and interactive input system 
1) HOST COMPUTER 

Referring to FIG. 15 and 16., host computer 9 preferrably 
comprises a digital computer system with high level image 
generation and graphics capabilities. A host computer 9 
compatable with the present invention is generally of the type 
manufactured by Silicon Graphics of Mountain View, CA as the 
Skywriter TM computer system. 

The high level 3-D image generation and 3-D graphics 
capabilities of computer system 9 typically consists of a type 
of digital computer subsystem which has* the capability to 
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texture -map at 1*3* frf-ifc. NTSC video feed Onto n 
three-dimensional wireframe 55.. The host computer 9. may 
include single or dual pipeline subsystem configurations. The 
computer 9. may include all signal processing means 3. 
comprising means 15, 16., 12, 18., 19., 1CL and 2_3. The high 
level image generation subsystem general ly includes processing 
means 17. for manipulation of the geometric model 1±, and 
processing means 18. for output of designated portions of the 
processed model 14_ for display. Computer 9_ includes a digital 
computer including at least one central processing unit, 
system bus, memory, communications ports, and the like. 
Computer 9 may be configured to drive one to twelve analog or 
digital output channels. Operating system software processing 
data, generated images, and the like are stored in mass 
storage devices which may include magnetic disk drives, 
optical disk drives, and so forth. Commands to operate the 
computer system 9. are entered by means of the . participant 
interaction devices 1_0. The computer 9_ may be configured to 
receive a single or plurality of inputs from a single or 
plurality of interactive input devices associated with the 
system 1JQ via host 9 communication ports. High- 
level 3-D image generation and 3-D graphics capabilities 
integrated with host computer 9. are generally of a type 
described in U.S. Pat. 4,827,445 by Fuchs; or manufactured by 
Silicon Graphics Inc., Mountain View, CA as Rea 1 i tyEng i ne TM 
Host Integrated Computer Image Generation System with VideoLab 
TM or VideoLab/2 TM input/output option, and with 
VideoSplitter/2 TM. option. Skywriter TM with Real ityEngine 
TM incorporates the IRIS Performer TM software environment to 
provide the performance and functional requirements for image 
gneration applications. 

The signal 5a. is typically captured from an image input 
system 6 by a conventional frame grabber at preferably at 30 
frames per second and converted from an analog to digital 
signal by a conventional analog to digital' converter 69 . 
Alternatively, the computer may recieve and operate on a 
multiplexed or compressed video signal*. The converted analog 
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signal representing the image is then transmi t t ted to the 
computer 9. for texture mapping on a 3-D wireframe 5_5 
representation defined in the host computer. Areas on the two 
dimensional frame £1 are referenced by look-up tables stored 
in memory of the computer 9_- Pixels on the two dimensional 
frame 61_ are assigned three-dimensional coordinates 
corresponding to a predefined wireframe, model 5_5 stored in 
computer memory 25a . The image segments are manipulated by 
the computer to generate the effect to the participant that he 
or she is within the computer generated environment. To 
accomplish this effect in the HMD assembly or large display 
assembly, the computer performs programmed mathematical 
operations on the input imagery. 

A mass storage device 25. may be a separate conventional 
magnetic disk device or can be an integral part of the host 
graphics computer %, i.e., the main memory of computer 9_. 
The mass storage device is used to store data representing 
beings, objects, and scenes representing subjects 1_3 in the 
world model 1_4. The mass storage device contains a previouly 
generated data base comprising a digital representation of the 
world model. FIGS. 11_ and 12 illustrate a top plan and side 
view, respectively, that can be termed the world model 1^4. or 
simulated environment constructed to illustrate the present 
invention. The subjects rendered in the model 14a are 
nominally divided into a plurality of convex polygonal 
surfaces 74_ having arbitrary numbers of sides. The respective 
polygons are represented in the data base as a sequence of 
data words each corresponding to a vertex of the polygon. By 
conversion, . the vertices of the polygon are sequenced in, for 
example, a counter-clockwise direction within the data group. 
A one bit flag in corresponding data word is utilized to 
indicate the last vertex. Each vertex data word suitabley 
comprises a plurality of fields representing the coordinates 
of the vertex in a choosen coordinate system (x, y, z), the 
intrinsic color intensities present at the vertex (R, G, B), 
and a vector indicating the. unit normal to the polygon surface 
at the vertex. In this manner the intensity and depth value 



2b 



WO 94/16406 



PCT/US94/00289 



o£ each pixel comprising an object is stored in computer 
memory . 

Alter nat ive ly, a single or plurality of live video feeds may 
be transmitted to computer 9 and processed in near real time. 
In such an instance, look up tables instruct the computer to 
sample predetermined portions of the image and texture map 
them onto predetermined portions of the 3-D wireframe model 
defined in the computer. Video compression systems and video 
multiplexer/demultiplexer systems may be operated to 
facilitate the reading in of images from a plurality of video 
sources. Or alternatively, a television production unit with 
image compression functions may be used to compress a 
plurality of sources into a single frame. A production system 
of a type and function compatable with the present system is 
described in U.S Pat. 5,130,794 by the present inventor. The 
preassigned area on the model 55 on which the image segment is 
positioned and oriented for texture mapping may be stationary 
or have motion. The model may correspond to a persons head or 
body or an object in the real world. In such an instance, the 
being or object is tracked by positions sensors located on the 
real object. In this manner the computer can keep track of 
the position and orientation of the corresponding model in the 
computer 9 and texture map each pixel image in its proper 
position on the model of the object or being. The ability to 
sample a live video feed is a requirement in telepresence 
applications of system 1, Real-time texture mapping from a 
live video feed is especially useful in an image based virtual 

teleconferencing system like that described in FIG 20. and FIG. 
21 . A television production system that can function as host 
computer 9, in system i especially designed for creating the 
effect of texture mapping live video on a 3-D shape is 
generally of a typed cited in U.S. Pat. 4,951,040 by McNeil et 
al., U.S. Pat. 4,334,245 by Michael, and U.S. Pat. 4,360,831 
by Kellar and corresponding products of a type generally 
available from Quantel Inc. of Darien, CT 'as the Digital 
Production Center with MIRAGE TM or ENCORE TM, a 3-D image 
processing system. 
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It will also be appreciated that only one participant may bt: 
receiving images from computer 9... And that either a singular 
or plurality of HMD 1 s assemblies 11 or large display 
assemblies 12. may be incorporated or interconnected. Or 
additionally, participants may use plural computers ( 9a to the 
nth) to operate on the same environment; e.g. 
telecommunications. In such an instance, only position and 
orientation data would be transmitted to a remote site to 
update the virtual model 14, of the remote computer 9_. Viewers 
may interact with one anothers virtual model instead of or as 
well as with virtual models generated by the computer 9. 

As shown in FIG. 1, preferably an integrated part of the 
host computer 9 is a graphics computer system 19.. The 
graphics computer system is in direct communicating 
relationship with the computer 9_. The keyboard, touch tablet, 
and host computer 9. are operated to control the processors and 
memory and other devices which comprise the graphics computer 
system. Various input sources may be routed to the graphics 
computer system for rendering. Once the graphics computer 
system has been operated to create or affect an existing 
picture, the picture is stored in mass storage or bused as a 
picture signal to the host computer image segment circuitry 
means, or directly to a display unit. The computer graphics 
system comprising a digital computer may be operated to create 
or effect the recorded video images either before or after the 
images have been texture-mapped onto a wireframe. Typically 

the participant 3-4 affects the video images frame by frame. 
Such a graphics system is used in the present invention to 
affect the images presented in either a 2-D or 3-D format. 
The data to be affected is derived from a video input source £ 
or storage device Z5. The components in a typical vector or 
raster electronic graphics system include a touch tablet, a 
computer, framestore, and a display. 

As shown in FIG. 1, it is foreseen that the system 1 may 
include an expert system 2_2 with a complementary data base. 
The expert system is in direct communicating relationship to 
computer 9_. The expert system may be'housed in a separate 
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chasis and communicate thru conventional conductors and 
input/output ports with computer 9. Or alternatively, the 
expert system may be integral to computer 9 . The knowledge 
system is provided to respond to participant 2_4 requests or 
actions, and each request or action has a record including a 
plurality of parameters and values for those parameters. The 
expert system is provided to process the record of a specific 
request or action to answer or respond to that request or 
action, and the complementary database stores a plurality of 
records of reguests or actions having known answers or 
responses, and any request from a participant 24 is 
preprocessed by searching the complementary database for a 
record identical to the record of the request or action. If 
an identical record is found, the known answer or response to 
the request having that identical record is given to the 
participant to answer his or her request; however if no 
identical record is found in the complementary database, the 
expert system is involked to answer or respond to the request 
or action. Experts system of a type generally compatable with 
system 1 are described. in U.S. Pat. 4, 884, 218 by Agnew et al . 
and U.S. PAT. 4,884,217 by Skeirik et al. Answers or 
responses transmitted from the expert system are interpreted 
by computer 9. such that the model 14 is updated. 
Additionally, responses transmitted from the expert system may 
be received by other peripherial devices 226 such as motion 
simulators, participant interactive tactile and force feedback 
devices, teleoperated vehicles, or any computer actuated 
device. Responses in the form of data is operated upon by the 
peripherial device to affect control surfaces, motors, and the 
like. In this manner subject beings modeled in the computer 
generated model may interact with subjects in the real world, 
environment. A motion simulator of a type compatable with 
system 1 device 228 generally responsive to participant 
actions and exper system data output is manufactured by 
McFadden Systems, Inc. of Santa Fe Springs> CA as the Model 
611A Motion System. A tactile and force feedback device that 
is generally of a type compatable with 'system 1 device 228 is 
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available from VPL Research Inc. of Redwood, CA as the 
DataGlove TM models THX 7 TSK, and FBX TM ' s . 

2) INTERACTIVE INPUT DEVICE SIGNAL PROCESSING 
An important part of signal processing means 3 is the 
interactive input system i£ operated by actions of the 
participant 24. to . manipulate the virtual beings, objects, and 
scenes within the virtual environment. Interactive input 
devices (e.g. data gloves with position, orientation, and 
flexion sensors, and HMD with position, orientation, and eye 
tracking sensors) are typically connected by RS-232 
input/output ports or the like to processing means of computer 
9 . 

Data from device sensors are typically translated to machine 
language compatable with software programming of the host 
computer by an interactive input systems 1 system electronics 
unit 7_8. The electronics unit 78. may be mounted adjacent to 
the device, a separate chasis, or in some configurations 
comprise a printed circuit board that is plugged into the 
system bus of computer 9_- Alternatively, an interactive, 
electronics unit 7_8 housing a single chasis may provide an 
interface between left and right data gloves, HMD, and voice 
recognition systems and computer 9.. The electronics unit 78. 
receives signals from the sensors over conductive cable and 
translate the signals into machine language compatable with 
the host computer 9. The electronics unit 7_8 contains 
interface circuitry to translate the data. The translated 
signals are transmitted by from the electronics unit to the 
computer 9. over suitable conductors. An electronics unit 7_e 
arrangement for interfacing various interactive devices to a 
host computer 9 workstation in system 1 is of a type generally 
utilized by NASA Ames Research Center, Moffett Field, CA, and 
operated as part of the Virtual Workstation and Virtual Visual 
Environment Display (VIVED) project. 

The preferred embodiment of the system 1 generally comprises 
two display means: a headmounted display (HMD) assembly 11 and 
a large display assembly 12.. Data from the input devices is 
operated upon by the processing means 'of computer 9 to affect 
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the virtual environment. Interactive input systems ICi and 
associated devices compatable with the present system 1 
include spaceballs, position sensor systems, pointing systems, 
'iatagloves, datasuites, voice recognition systems, and other 
such input devices. Electro-optical, ultrasonic, and magnetic 
position sensing systems for tracking an object or being are 
also known to those skilled in the art and may be incorporated 
with the present system 1. Position sensing systems worn by a 
viewer or mounted on an object of a type particullary 
compatable with and that may be incorporated into the present 
system 1 include those manufactured by Ascension Technology 
Corporation of Burlington, VT as A Flock of Birds TM, , and 
Polehemus of Colchester, VT as 3Ball TM, 3SPACE TM Tracker, 
and 3SPACE TM I SOTRAK TM. U.S. Pat. 4,542,291 and 4,988,981 
by Zimmerman et al. describes an interacitive input device and 
method compatable for incorporation with the present system 1. 
Products consistant with Zimmeran's invention are available 
from VPL Research Inc. of Redwood City, CA, as the DataGlove 
TM, DataVest TM, and DataSuit TM product lines. Associated 
VPL software compatable with VPL interactive input system 10, 
including associated electronic units, interface circuitry, 
devices, and with computer 9 (e.g. Skywritter TM workstation 
with RealityEngine TM image generation subsystem) is provided 
for incorporation with system 1. The VPL input device 10 
contain a microprocessor that manages the real time tasks of 
data acquisition and communication through the various RS232C, 
RS422, user port and optional IEEE 488 port. The 
microprocessor also controls a 3Space Isotrack TM position and 
orientation system incorporated in the control unit 1£. Host 
computer 9_ is programmed with software that allows a library 
of gestures, solid model simulations of complex objects, and 
the manipulation of said objects in real time. 

Additionally, non-contact position sensor systems 97_, such 
as electro-optical systems, described in U.S. Pat. 4,843,568 
by Krueger et al., U.S. Pat. 4,956,794 by Zeevi et al. are 
comptable and may be incorporated with the present system 1. 
The radar in U.S. Pat. 5,005,147 by Krishen et al- and the 
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LADAR available from Autonomous Technologies Corp. of Orlando, 
FL may also be positioned above the viewing space to sense the 
position and orientation of participants and objects in the 
viewing space 58.. Data from the radar or LADAR is processed 
by suitable computer processing means to determine the 
position and orientation of the viewer in the viewing space. 
The position data is then processed by computer 9. to affect 
the model 14_ of system 1. Additionally, data from the same 
sensors that track the position and orientation of a target 
subject 13 (e.g. LADAR or radar system with a video camera) 
may be operated upon by computer 9 to reconstruct the subject 
as a model 14a . In such an instance input system 2 and 
position sensing system 10. constitute the same system. The 
combined system 2. and 10. is placed about the viewer as in FIG. 
2 . 

Furthermore, as shown in FIG. 17 , a voice recognition system 
227 may operate to translate audible sounds made by the 
participant 24_ into machine language to control processing 
functions of the computer 9_ . The voice recognition system 
includes a microphone 39_ worn by the participant that 
transmits a signal representing audible sounds over an 
associated conductor with an input jack arranged in receiving 
relationship to the voice recognition printed circuit board. 
Alternatively, a conventional over-the-air radio frequencey 
transmitter and associated electronics are located in 
communicating relationship with the microphone. In this 
manner the particiants voice signals are transmitted 

over-the-air to a corresponding receiver in communicating 
relationship to the voice recognition system. A board level 
voice recognition system of a type utiized in system 1 is 
available from Speech Systems, Batavia, IL, as the Electronic 
Audio Recognition System. The boards may be mounted in the 
rack of the host computer ±, or in a separate chasis in 
communicating relationship with computer S_. The voice 
recognition system 227 operates with computer 9. to convert the 
voice signals into machine language to affect the model 1± of 
system 1. . * 
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The monitored information from input system 10. via 
conductors 82. is used to modify standard computer graphics 
algorithms of the computer 9. This preferably comprises a 
laser-disc based or other fast-access digital storage graphic 
imagery medium. Typically, the viewer of either the HMD 
assembly or large display assembly may interact with the 
virtual model 14_ by issuing suitable commands to the computer 
9. by manipulative means or sensors attached to his 
fingerste.g. DataGlove, TM) . Based on such manipulation, the 
spatial coordinates of the virtual model can of course be 
changed to give the impression of movement relative to the 
viewer . 

3) IMAGE PROCESSING FOR DISTRIBUTION AND DISPLAY 
Once the computer 9_ has updated the world model 14_ a given 
field or fields of view are selected in the computer and 
transmitted to each display unit TjO or units ( 70 to the nth), 
and each audio speaker 3_5 for presentation to a participant. 
Display units 70. may consist of conventional 3_1, stereographic 
32 or autoster eogr phic 3_3/ oz holographic 3_4 television or 
projection display units. Typically, a single or two views 
are sampled by the computer for display on the HMD 1J^. 
Typically six adjacent and independent views are sampled by 
the computer 9. for display on the large presentation assembly 
12. 

In FIG. 15. thru 18. computer 9. includes a video processing 
means 1S_, such as a "VideoSpli tter /2 1 ' TM printed circuit 
board, that is programmed to select six independent and 

adjacent fields of view of the world model 14a for display on 
the large display assembly 12.. The video processing means is 
in communicating relationship with the display processor and 
raster processor printed circuit boards. The raster processor 
comprises a board level unit that includes VLSI processors and 
graphics system memory. , The data received by the raster 
processor from the output bus of the geometry processor 3J7 
printed circuit board and data management system of computer 9_ 
is scan-converted into pixel data, then processed into the 
frame buffer. Data in the frame buffer is then transmitted 
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into the display processor of computer 9_. Image memory is 
interleaved among the parallel processors such that adjacent 
pixels are always being processed by different processors. 
The raster processor contains the frame buffer memory, texture 
memory, and all the processing hardware responsible for color 
allocation, subpixel anti-aliasing, fogging, lighting, and 
hidden surface removal. The display processor processes data 
from the digital frame buffer and processes the pixels thru 
digital-to-analog converters (DACs) to generate an analog 
pixel stream which may then be transmitted over coaxial cable 
to display units 3i, 32, or 33. as component video. The 
display processor supports programmable pixel timings to allow 
the system to drive displays with resolutions, refresh rates, 
and interlace/non-inter lace characteristics different from 
those of the standard computer 9. display monitor. The display 
processor has a programmable pixel clock with a table of 
available video formats (such as 1280 x 1024 at 60 Hz 
Non-interlaced NI or VGA (640 x 497 at 60 Hz NI ) , NTSC, PAL, 
and HDTV). All printed circuit boards comprising the forth 
processing means 18. may be held in the chasis of the computer 
9 and generally of a type such as the Real i tyEngine TM host 
integrated computer system, and including the IRIS Performer 
TM software environment available from Silicon Graphics, Inc. 
of Mountain View CA. 

Each of the six fields of view selected by computer 9_ 
correspond to imagery representing a 90 degree square field of 
regard. Corresponding square fields 71a-71f of regard are 
displayed adjacent to one another and form a cube such that 
imagery representing a continuous 360 degree field of view 
scene is displayed about the participant. Video signals 
representing each of the six fields of view are transmited 
over a respective channel of the video processor 2A to an 
appropriate dislay unit 70a-70f or corresponding image segment 
circuit means 72. for additional processing. 

FIG. L7 illustrates an embodiment of system 1, in which image 
segment circuit means 7_2 includes image control units 73a-73f 
which operate to distribute the images, compr is i ng spherical 
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coverage to the large display assembly Y2. The function of 
the digital processing circuitry means is to accept an 
incoming television signal and to display it over an array of 
display units 11_ so that the resulting image appears as it 
would on a single very large screen TV that completely and 
continuously surrounds the viewer. 

In FIG. 12 computer 9 transmits each of the six fields of 
view is transmited over a respective channel of the 
"VideoSplitter " TM to a corresponding image control unit 
73a-73f . The electronic video control system 72a-72f accepts 
the incoming video signal and partitions and processes it for 
display on the display units 70al-70f 9 . This partitioning is 
referred to as segmenting the image. Each respective image 
control unit processes its repective image into image segments 
71al-71f 9 . Each segment is transmitted from the image control 
unit to predetermined adjacent display units such that a 
continuous panoramic scene is displayed about the participant 
on the display units of the large display assembly. 

Within each image controller 73a-73f is a central processing 
unit (CPU) which executes software commands to effect the 
processing of the images in each framestor memory of 
framestore cards. The CPU's of each image controller are 
connected by an internal bus to each respective framestore 
card. The software commands read by the CPU may be 
pre-recorded onto optical disk, videotape, or image controller 
by use of a conventional microcomputer or computer workstation 
9-. The microcomputer or computer workstation preferrably 
includes a keyboard for an operator to enter software commands 
to the image controllers to effect image display. Software 
commands consist of time code and program code for control of 
the displayed picture. 

The microcomputer or computer workstation can also be operated 
to input software commands that specify the picture aspect 
ratio to be scanned for display. In this manner the signals 
representing model 14a may be segmented by 'the image 
controller into image segments. The video signals 
representing each framestore cards picture segment is then 
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transmitted to a corresponding display unit 70.. The number of 
picture segements that the image controller can segment the 
composite picture into varies and determines the maximum 
number of display units that can be accommodated. 
Preferrably, image segments are pre-formated by the earner? 
system 6_ or computer 9_ processor IB to correspond to the 
picture segmentation accomplished by the image controller. 

Image segment circuit means 12., including assoicated cables, 
display units, audio system, and so forth of the type 
generally utilized in FIG. 11 of the system 1 is marketed by 
DELCOM USA, Philadelphia, PA and includes image controller 
units 73a-73f sold as the Station Pro and Universal; and 
systems from North American Philips Inc., of New York, NY 
under the name VIDIWALL TM. 

Other multiple display unit wall arrays that may be 
incorporated to form assembly 12_ within the present system i 
include that described in U.S. Pats. 5,010,413 by Bahr, Inc., 
U.S. Pat. 4,890,314 by Judd et al., U.S. Pat. 4,974,073 by 
Inova, U.S. Pat. 4,974,073 by Inova, U.S. Pat. 5,016,109 by 
Gaylord, U.S. Pat. 4,734,779 by Levi et al., and the MediaWall 
TM available from RGB Specturm of Alameda, CA, Vidiwall TM 
available from Phillips of Holland, TeleWall Delcom 256 model 
available from Nurnberger Medientecnick GmbH of Germany, and 
VideoWall TM available from Electronic of Minneapolis, MN . 
Large display assemblies compatable with the present invention 
are described in detail in U.S. Pat. 5,130,794 by the present 
inventor. 

As disclosed by Hooks, U.S. Pat. 4,463,380 and in numerous 
other prior art, the perspective of the beings 14a , objects 
14b , and scenes 14c displayed to the participant 2_4 are 
calculted to provide a correct perspective appearance to the 
participant. To achieve a participant-centered perspective, 
in either the HMD assembly LI or large display assembly 12 , 
off-axis perspective projection is calculated. Off-axis 
perspective is calculated based on the position and 
orientation of the participants head and eyes 75.. The 
simplest derivation alters a standard on-axis perspective 
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L---y?.t*OT:iOri by performing two mathematical transformations of 
model 14a coordinates that are the equivialent to. the 
translation, contraction, and expansion with respect to the 
fixed origin and fixed coordinate system. First, points are 
sheared in a direction parallel to the projection plane, by an 
amount proportional to the point's distance from the 
projection plane. Then, points are scaled along the axis 
perpendicular to the projection plane. In this fashion, 
perspective is calculated by the computer means 18. based on 
the participants position anywhere in the viewing space. Not 
only is the perspective distorted based upon the participants 
location in the viewing space realitive to the virtual model 
14a , but imagery is also distorted to compensate for the angle 
from which the viewer observes the imagery on each display 
unit 7_0 to the nth. For both reasons, perspective may be 
grossly distorted to make the subject or scene appear natural 
from a participant-centered perspective. 

Furthermore, as disclosed by Waldern, U.S. Pat. 4,984,179 
and in numerous other prior art, movement of the virtual model 

I, 4 may be made relative. A slight movement of the viewers 
head may be translated by the computer to move beings, 
objects, or the scene represented as the virtual model only 
slightly to dramatically based on algorithms preprogrammed 
into the computer 9_. 

4) PROCESSING SYSTEM EMBODIMENTS 

In FIGS. 15. input means £ transmits data to either computer 
and/or 2Ja. Computer 2a. transmits signals to a HMD assmbly 

II. Computer 9b. transmits signals to the large display 
assembly 12.. Participant 24a and 24b each operate interactive 
devices of associated with system 10a and 10b , respectively, 
to transmit position data to their respective computers 9a_ and 
9b. Position and orientation data from either participant may 
be transmitted by conventional interface means between 
computers 9a, and 9Ji/ 0E interactive input systems 10a or 10b 
to computer 9a, and 9b. in order to update the computer 
generated model 14.. Alternatively, both participants may 
operate their respective computer independently and not be 
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networked together in any manner. 

Alternatively, as shown in FIG. a single computer 9 in- 

operated to render a computer generated model for both the 
viewer of the HMD assembly 11 and large display assembly 12,. 
Each participant 24a and 24b operate interactive input 
systems 10a and 1 Ob respectively devices to effect the model 
11- 

Assume for example, the 3-D digitizing system 1_ comprises a 
systems electronic unit, keypad, footswitch, stylus, and model 
table. The system electronics unit contains the hardware and 
software necessary to generate magnetic fields, compute the 
position and orientation data, and interface with the host 
computer 9_ via an RS-232C port. The keypad is a hand-held, 
multikey alphanumeric terminal with display that is used to 
command data transmission, label data points, transmit 
software commands, and receive and display host computer 9_ 
messages. The foot-operated switch is used to transmit data 
points from the digitizer to the host computer 9.- The stylus 
is a hand-held stylus that houses a magnetic field sensor and 
is used to designate the point to be digitized. The operator 
places the stylus tip on the point to be digitized and 
depresses the foot switch or keypad to collect x, y, z 
coordinates of an object, being, oi to define the background 
of the scene that constitutes the subject. The points are 
then selectively connected by lines within the computer 3_ to 
construct 3-D wireframes. The 3-D wireframes 55 form a data 
base which is stored in the memeory 25a of the computer 9.- 
The model table is a free-standing 40" high model table used 
as the digitizing surface. The table houses the electronics 
unit and the magnetic field source (transmitter). The volume 
of space above the table is specifically calibrated at time of 
manufacture and this data reside in the electronic units 
memory. A type of 3-D digitizing system utiilized is 
generally of the type manufactured by Polhemus Inc. of 
Colchester, Vermont as the 3SPACE Digitizer. 

FIG. r3 illustrates a subject vase 13. to be modeled in the 
computer 9.* The computer is configured to recieve input 
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imagery 5b_ and shape data 55. from the input sources 2. 
Subject imagery 5G_ of the vase recorded by the panoramic 
camera system 6. and vase shape data derived from operating the 
digitizer system 2 are combined by operating the computer 9.. 
As shown in FIG. 8b., 9b, and 1£ the computer is operated to 
texture map image segments of the recorded image of the vase 
onto 3-D wireframe 55 of the vase previously constructed by 
operating the' 3-D digitizing system 7. Once the texture 
mapping rendering is accomplished each rendered object(e.g. a 
vase), being (e.g. a person), or scene (e.g. temple and 
seascape) is placed in the computers memory 25a as rendered 
models 14a . A type of video input and output sub-system for 
combining 3-D graphics with digital video as described above 
is manufactured by Silicon Graphics of Mountain View, CA under 
the name VideoLab2 TM. The system allows digitized directly 
to the screen at real-time rates of 25 or 30 frames per second 
or stored on disk in memory of mass storage 25a . 

Concurrent with shape and image processing, the accoustical 
signatures 5c of a subject 13_ are recorded. FIG . 1_3 
diagramatically illustrates a sample accoustical signature of 
a vase. The accoustical signature of the vase is derived by 
manipulating the vase in the real world and recording the 
sounds the vase makes. For instance the vase may be handled, 
scratched, hit, or even broken in order to render sounds by 
various accoustical signatures . Those signatures form the 
audio world model 14b . Accoustical signatures are linked to 
subjects in the visual world model 14b. The accoustical 

signatures recorded by microphones 39a-39 f of the audio input 
system 8. are converted from an analog signal to a digital 
audio format and may be stored in the audio processing means 
mass storage 25b . Alternatively, audio storage 25b may 
consist of accoustical signatures stored on conventional audio 
tape or disc and accessed from corresponding audio tape and 
disc players. Mass storage 25b is connected by suitable 
conductors and in communicating relationship with audio 
computer processing system 2_3. When an. action, such as a 
person breaking the vase occurs in the world model 14a occurs, 
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a corresponding audio representation is pre-programmed to 
occur in the accoustical world model 14b . The stereo audio 
signal is typically converted from a digital to stereo analog 
signals and read out. to the stereo headphones in the 
participants HMD 11 or to speaker systems associated with each 
side of the large display assembly 12. 

FIG. 14_ diagramatically illustrates a computer generated 
world model 1J_ which the viewer can enter by the use of 
interactive input system 10, and audio-visual means 4_. The 
system comprises imagery, shape, and audio signatures recorded 
from objects in the real world and then translated and 
represented in the computer generated world. By combining 
position and orientation data received via conductors TS (for 
the head sensor) and El and Er. (for the left and right eye 
sensors) the computer 9_ determines the current field of view 
for the participant 24. and his or her line of sight. The 
images transmitted to the display unit's 70. to the nth are of 
a virtual model 14_ having predetermined spatial coordinates 
within the display assemblies. With the movement of each 
position sensor relative to the wearer and with each variation 
of the direction in which the sensor wearer is looking data is 
transmitted to the computer 9. via conductors 82a and 8 2b . 
Whereby two images 80g and 80h are received over conductors 
for display on the HMD 11. and six images 8Qa-8Q f are 
transmitted over conductors for display on the large display 
assembly 12.. Only the portion of the texture mapped imagery 
comprising the 3-D model 14. within the helmet wearer's 24. 
field of view is transmitted to the HMD's 1JL. Two images are 
calculated so as to present a stereoscopic view of the virtual 
model 14a cor r esspond ing with what would be seen if the 
virtual model were a real object from the helmet wearer's 
current standpoint. With respect to the large display 
assembly 12 the participant observes imagery within his or her 
actual field of view as displayed by the six display units 
81a-81f . The six display units display a /virtual model 
corresponding with what would be seen by the viewer if the 
virtual model were a real object from 'the viewers current 
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standpoint . 

Still refering to FIG. 1±, the location and angular position 
of the participant's 24. line of sight are calculated by 
computer 9. Three-dimensional algorithms are operated upon so 
that the observer is effectively enclosed within the model 
14a.. Typically the model is generated by rendering data word 
or text files of coordinates comprising the vertices of the 
surface faces which are form the model 14a . With the 
participant at a known position relative to both the model 
14a-b( the vase) and the center and the three-dimensional world 
11 which is to be occupied by the participant the angular 
orientation of the participant can be derived using basic 
trigonometry. The projection of the model is made by 
computing the persepective position of a point and 
subsequently drawing it on a display unit. 

Referring again to FIGS. 15 and 16, computer 9 transmits 
processed signals to the HMD assembly 11 which the viewer 
wears. An important part of signal processing means 3 is at 
least one interactive input device 10.. Device 1_0 continuously 
monitors the participant. The position system 10 comprises a 
head sensor 76a, a right data glove sensor 76b worn on the 
participants right hand Hr_, a left data glove sensor 76c worn 
on the participants left hand HI, a source unit 77, a systems 
electronics unit 78., and conductors 8_3, and 8± that 
interconnect the. system. The source transmits a acts as a 
fixed source module and is energised by a suitable power 
source and transmits a low frequency electrical field. The 
sensors are small and modular. The head sensor 76a is mounted 
on a band worn on the viewers head, and the glove sensors 76b 
and 76c are mounted intragral to the right and left glove, 
respectively. The sensors sample the field generated by the 
source. The sensor and the transmitter are both connected to 
an electronics decode circuit 78. which decodes the signals 
from the sensors and computes each sensor's position and 
orientation in angular coordinates. The device thus provides 
data as to both the position and orientation of the viewers 
head, left and right hand. The information is passed via 
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conductor S2 to the computer 9. A type of system generally 
compatable with the present invention is manufactured by 
Polhemus of Colchester, VT 7 as the 3SPACE TM Tracker and 
Digitizer. Additionally, th^ir may be built into the head 
band two eyeball movement tracking systems, or eye sensors 
which monitor movements of the wearer's eyeballs and transmit 
signals representing these movements to computer 9_ via 
condutors. A type of eye tracking system generally compatable 
with the present invention is manufactured by NAC of England, 
UK as the Eye Mark Recorder, Model V (EMR-V) . 

A slightly different position sensing arrangement is 
incorporated for the large display assembly 12 in that the 
participant wears a head band which holds a position sensor 
76a. Other than that, the position sensor system 10a mounted 
on participant 24b in assembly 12 operates in the same manner 
as the position sensing system 10b operated by participant 24b 
wearing the HMD assembly 11.. 

In FIG. 16_ computer 9. includes computer image processing 
means 18 for sampling out up to eight independent views within 
the model. 1± for video output. Typically, each output signal 
is transmitted to a display unit 70a-70h . The circit means 18 
samples out two independent views 70g and 70h for display on 
the left and right displays of the viewers HMD 11, and samples 
out six independent contiguous views 70a-70f for display on 
each side of the six sides of the large display assembly L2. 
It should be noted that by decreasing the number of displays 
the computer 2. and associated circuit means UL is able to 
increase the display resolution because more processing memory 
can be generated for the remaining displays. A type of image 
segment circuit means 18. compatable with and integral with the 
present invention is generally of a type manufactured by 
Silicon Graphics Inc. of Mountain View, CA under the name 
VideoSplitter/2 TM. 

Fig. 17 is a more automated embodiment of the system 1 in 
which a 3-D subject is rendered by a first/ processing means 15 
for fusing the image signals 5a and microwave signals 5b from 
the subject 13.. The fusion of the ima§e and microwave signals 
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by means 15_ result? in. 3-D model segments 26a . A panoramic 
3-D camera system £ and panoramic 3-D digitizing system 7_ 
comprises the panoramic input system 2_. Arrays 4_0 and array 
assemblies A± similar to those shown in FIGS. 4_ thru 1 are 
preferrably incorporated position and hold sensors in place. 
A plurality of image and microwave sensors are positioned 
inward about a subject 12 to form adjacent coverage 4 2 . 
Likewise, a plurality of image and microwave sensors are 
positioned in an outward facing manner to achieve adjacent 
coverage 4_2 of a subject. A target 3-D subject exists in 
space in the real world. The outputs of image and microwave 
sensors are combined by image processors lj? and 1£ of computer 
9.. The image sensor is composed of one or more cameras 3_7 
which consist of an aperture and a light sensitive element 
such as a charge-coupled device (CCD) array 53.. The light 
sensitive element converts the sensed image into an analog 
video signal 5a^. The analog video signal is transferred to 
the low-level image processor of means 15_ via a coaxial cable. 

The low-level image processor is. an electronic circuit that 
may be implemented as a general purpose computer with 
specialized programming or as a series of specialized circuit 
boards with a fixed function. The low-level image processor 
collects the analog video for each frame and converts it to a 
digital form. This digital image is an array of numbers 
stored in the memory of the low-level image processor, each of 
which represents the light intensity at a point on sensing 
element of camera. The low-level image processor may also 
perform certain filtering operations on the digital image such 
as deblurring, histogram equalization, and edge enhancement. 
These operations are well-known to those skilled in the art. 

The digital images, thus enhanced, are transferred via a 
digital data bus to the high-level image processor of means 
1^.. The high-level image processor may also be either a 
hard-wired circuit or a general purpose computer. The high, 
level image processor takes the enhanced digital image and 
attempts to extract shape information from it by various means 
including shape-f r om-shading, or in the case where multiple 
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cameras are used, stereopsis cr phot omet er ic stereo. Again 
these are well-known operations to those skilled in the art. 
When multiple cameras are used in the image acquisition and 
processing system, the multiple images are combined at the 
high-level image processor. Alternatively, the images may be 
combined optically by the use of a panoramic camera system 
shown in FIG. 3,, and previously discussed herein. The image 
processor processes an incomplete surface model of the unknown 
object. The surface model is a list of digitally stored data, 
each of which consists of theree numbers that are the x, y, 
and z locations of a point on the object's surface. The 
incompleteness of the surface model may result from regions on 
the object's surface that are, for some reason, not understood 
by the high-level image procesor. The incomplete surface 
model is passed on to the initializer included as part of 
processing means 15 by a digital data bus. 

The initializer is a general-purpose computer of digital 
circuit that "fills in" the portions of the surface model left 
incomplete by the high-level image processor. The unknown 
areas of the surface model are computed by surface functions 
such as B-splines that depend on some numerical parameter p. 
The surface functions are represented digitally in a form that 
both the initializer and the computer understand. The surface 
funcitons along with the complete surface model are passed on 
to the computer by a digital data bus. The computer will 
determine the correct value of the paramater p in the manner 
hereafter described. 

Concurrently with collection of the images by cameras, a 
radar cross-section (RCS) of the unknown subject 13_ is being 
measured by the radar system 28,. The radar 3_£ functions a 3-D 
shape input system 7.- The radar 38. consists of a radar 
processor, antennas 85 , and waveguides 86.* The radar 
processor is a widely available device, and all of the 
functions described here can be performed by a unit such as 
the Hewlett-Packard 8510 Network Analyzer. The method by 
which they may be performed is described in Hewlett-Packard's 
product number 118510-2. The radar processor generates a 
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microwave signal that is transmitted along the wavguide and 
radiated, by the transmitting antenna. The electromagnetic 
field is diffracted by the object and collected by the 
receiving antenna. The diffracted signal is transmitted back 
to the radar processor by a waveguide; The radar processor of 
system 2 computes the RCS of the unknown subject 13. The RCS 
is represented digitally by the radar processor, and 
transferred to the computer processing means 15 by a digital 
data bus 5b . 

The computer 15. performs the comparisons and interations 
using two pieces of widely available software. The first is 
the MINPACK package of non-linear minimization programs 
published by Argonne National Laboratory, and the second is 
the Numerical Electromagnetic Code (NEC) available from Ohio 
State University. The NEC code generates theoretical 
approximations of the RCS of the object using the surface 
model produced by the initializer. The MINPACK program 
"lmdifO" uses these approximate RCSs to compute the correct 
value of the parameter p by an iterative scheme known as 
"nonlinear least squares". This method allows computation of 
the correct value of p by minimizing the differences between 
the observed RCS acquired by the radar system and the 
theoretical RCS computed by the NEC code from the incomplete 
surface model. Using the correct value of p, along with the 
incomplete surface model and surface functions from the 
initializer, the computer 15 generates a complete surface 
model segment for each adjacent field of regard 4-2- of the 
overlapping 41. radar and image sensors. Preferrably, a 
plurality of means 15. operate in parrallel to derive segments 
26. representing all sides of a 3-D subject 13. simultaneously. 
As with all computer programs, they must be written for the 
particular hardware and carefully debugged. U.S. Pat. 
5,005,147 by Krishen et al. discloses a "Method and Apparatus 
for Sensor Fusion" generally of a type that is compatable with 
the present invention herein referred to as the sensor fusion 
system that forms the first processing* means L5. 

The output from each of the sensor fusion systems is 
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transmitted to. a solid-modeling computer system which forms 
the second processing means 16. of the system 1 shown in FIG. 
12. The computer system includes a high speed central 
processing unit (CPU), one or more terminals, mass storage, 
for instance magnetic tape or disk drives,, a solid-state 
memory, one or more communication interfaces (such as a 
network interface), a high resolution video display and a 
printer plotter. 

Mass storage of means 16. contains a database for storing 
solid models representing segments 26. of subjects 13. such as 
objects, beings, and scenes. Solid models may be represented 
in the database in a number of different ways, such as in the 
form of boundary representations, and faceted representations. 
Preferrably, solids are represented so that it is possible to 
extract information about surface patches whose disjoint union 
comprises the bounding surfaces of the solid. 

The system operates using topology directed subdivision for 
determination of surface intersections. This is accomplished 
by the steps of obtaining a pair of surfaces from a main pool 
of surfaces representations and performing a mutual point 
exclusion test to determine if the surfaces may have an 
intersection. Data defining the model segments 26. obtained by 
the first processing means 15 are transmitted to the second 
processing means 16 mass storage. Included in this data is 
information on the surfaces representations of each model 
segment which is operated upon by the second processing means 
in determining surface matching and intersection. For those 
pairs of surfaces possibly having an intersection, the 
transversality of the surface is checked. If transversal, the 
intersection set is computed. For those pairs which are not 
transversal, recursive subdivision is performed until 
transversality is established or until a flatness criteria is 
met. A parallel processing system including a master 
processor and a plurality of slave processors performs the 
subdivision operation on the surfaces in a parallel fashion. 
The bounded surfaces are then processed as completely rendered 
3-D objects, beings, and scenes. The resultant data defining 
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and representing the 3-D objects, beings, and scenes is read 
over a conductor into the memory 25a of the third processing 
means r7. U.S. Pat. 5,014,230 by Sinha et al . discloses a 
"Solid-Modeling System Using Topology Directed Subdivision for 
Surface Intersections'* generally of a type compatable with the 
present system 1 and referred to herein as a solid-modeling 
computer system that comprises the second processing means 16 . 

In FIG. IJ7 a position sensing system 10. similar to that 
described for use in FIGS. 2 and 3 is operated to monitor the 
participants position and orientation. However, it is known 
to those skilled in the art that various viewer interactive 
input systems, such as those previously decribed, may be 
incorporated to monitor the position and orientation of the 
ve iwer . 

In FIG. 17. audio signals 5C from the microphones 39a-39f are 
transmitted to the 3-D audio system 23.. The audio system 23 
outputs a stereo signal 5c. corresponding to the subjects in 
the model 14a and the position and orientation of the 
participant 24_. The stereo signal 79. is transmitted by a 
conventional stereo audio transmitter 6_2 to a audio stereo 
reciever 63. associated with stereo headphones 6_4 worn by the 
participant 24.. Preferrably .signal 79. is transmitted 
over-the-air so that no conductor cables or wires incumber the 
participant 2_£. Such a stereo transmission/receiver headphone 
arrangement is of a conventional nature used commonly by those 
in the broadcast industry. 

In FIG. storage and geometric manipulation of the 
panoramic computer generated model 14. by computer 9. is the 
same as in FIG. 16, for the large display assembly. However, 
the forth processing means 18. for processing the image for 
display and distribution also includes image segment circuit 
means 72, to partition each signal 80a-80f into sub-segments 
for display on an array of display units 70al-70af located on 
each side of the viewer. Image control units 73a-73f , whose 
detailed operation has been previously described, are included 
as part of means 7_2 to process the images for distribution to 
the display units. 
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d) THREE-DIMENSIONAL DISPLAY PROCESSING 

Referring to FIGS. 18. and 19, it will be appreciated that 
some embodiments of input sources 2 and signal processing 
means 3 allow a three-dimensional model 14a to be rendered 
within the computer 9. that is able to be rendered in real-time 
for simulation. Additionally, the model has increased realism 
because the model is composed directly from shape and imagery 
recorded from the real world as opposed to created in a 
traditional manner incorporating computer graphics. Most 
importantly, computer 9. is programmable such that multiple 
viewpoints of the three-dimensional model 1± may be processed 
out to facilitate dynamic three-dimensional display for either 
stereographic, autostereographic or holographic display system 
applications 3±. Computer 9_ includes the necessary firmware 
and software for processing and distributing 3-D images to 3-D 
display units 32, 33., or 34.. 

In such applications, a single computer is preferrably used 
to process an adjacent segment 26a of the model 14a for each 
3-D display unit .32, 33., or 3±. This is recommended because 
of the computational intensive nature of rendering multiple 
viewpoints of the model. The complete world model 14a may be 
stored in the memory of each computer 9a to the 9nth . 
Alternatively the system may be configured such that each 
computer 9 recieve only a segment 26a of the model 14a from a 
master computer 9. or data base 2 5a . Preferrably a participant 
interactive input system 10_ transmits viewer position and 
orientation data to each computer via conventional 
interfaces 82.. Each computer is synchronized with each of the 
other computers by a master clock 8£ transmitting a timing 
signal to each computer 9a^ to the 9nth . The timing signal may 
be derived from the internal clock of any computer 9.. The 
output signals from each computer 9al-9f 9 are transmitted to 
each respective 3-D display unit 90al-90f 9 of the display 3-D 
assembly. The display units display adjacent portions of the 
model 26al of the world model 14a such that a substantially 
continuous stereographic, autostereographic, or holographic 
scene 71 is rendered before the part icipants eyes . 
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With respect to stereoscopic methods and display systems 3 ~: 
compatable with system 1 which require special eye glasses; 
commonly, two overlapping pictures for each screen can be read 
out for 3-D stereoscopic effects, such as with differing 
filtration or polarization between the projection modules or, 
for a 3-D effect without glasses, showing one version of the 
picture slightly smaller and out of focus relative to the 
other. The two differing points of view are projected in a 
super-imposed fashion on each side of the large display 
assembly 12.. The images are scanned out of computer 9. as 
separate channels by using a multi-channel video output 
processor 18. generally of a type aviailable from Silicon 
Graphics Inc. of Mountain View, CA as the VideoSpl i tt er /2 . 
Spatial separation is achieved because each of the images 
processed out of computer S_ is taken from a different point of 
view. Polarizing film is placed in front of the projectors 
70a and 70b , one horizontal and the other vertical, and the 
polarized image is viewed with polarized eye glasses of the 
same orientation, resulting in each eye seeing the proper veiw 
as with a stereo viewfinder. The number of projection units 
would remain the same as the number of points of view. The 
distance between the point of view of the images scanned out 
of the computer 9. by the multi-channel processor would be 
approximately 6 cm, equivalent to the average distance between 
the pupils of human eyes. 

Alternatively, a multi-mode stereoscopic imaging arrangement 
according to U.S. Pat. Applications SN 7/612,494, SN 
7/536,419, SN 7/561,090, and SN 7/561,141 by Faris is 
compatable and may be integrated with system 1_. Multiple 
adjacent viewpoint images of the model, preferrably of the 
left and right eye, are spatially multiplexed. The 
multiplexed image is applied to a micro-polarizer array. The 
multiplexed image is demultiplexed by the viewer wearing 
polarized eye glasses. An advantage of Far is 1 s system is that 
a single picture frame contains both left and right eye 
information so that conventional displSy units are instituted, 
but yield a stereoscopic image to the viewer. 
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Alternatively, an autostereoscopic image requiring no eye 
glasses may be rendered by scanning out multiple adjacent 
viewpoint images of the model from computer 9 and applying 
those images to processing meams 119. and display mean? 32 
according to U.S. Pat. 4,717,949 by Eichenlaub or an 
autostereoscopic full-color 3-D TV display system 33 
incorporating a HDTV LCD color video projector and bi-plano 
3-D screen with diffusion plate and lenticular screen 
available from NHK Science and Technical Research Labratories 
of Japan. Both Eichenlaub and NHK employ multiple 
interdigitated columns of left and right eye images in a 
corduroy-type manner. The images may be interdigitated by 
computer 9. Alternatively, the images may be read out from 
computer 9 on separate channels and interdigitated by a signal 
mixer. 

Alternatively, an autostereoscopic system includes a system 
in which multiple adjacent viewpoint images of the model 26a 
of the world model 14a are applied to processing means 119 and 
display means 33 according to U.S. Pat. 5,014,126 by Pritchard 
et al. The scanning path and recording rates are selected to 
produce motion within visio-psychological memory rate range 
when displayed using standard diplay units 31. Computer code 
is written to define the multiple adjacent viewpoint images of 
the model 14 and the number of frames per second that the 
computer 9 transmits to the display unit. 

Finally, holographic systems compatable with system 1 
according includes U.S. Pat. 4,834,476 by Benton which 
discloses a method and devices 33 for display and sampling a 
series of perspective views of the model corresponding to the 
participants 25 position that can be computed by conventional 
computer graphic techniques. Each computer 9al-9nth is 
programmed and operated to construct a series of predistorted 
views of a model segment 26a of the world model 14a from image 
data stored in a computer memory by ray-tracing techniques. 
Each perspective view can then be projected with laser light 
onto a piece of high resolution film f&om the angle 
corresponding to its computed viewpoint, overlapped by a 
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coherent "reference" beam to produce a holographic exposure 
that records the direction of the image light. After all the 
views have been recorded in this way, the hologram is 
processed and then illuminated so that each view is sent back 
out in the direction, it was projected from, that is, toward 
its intended viewing location, so that a participant moving 
from side to side sees a progression of views as though he or 
she were moving around an actual object. Each adjacent 
holographic display unit 34al-34f 9 displays a holographic 
image of an adjacent corresponding portion 26a of the world 
model 14a . 

For some applications, a holographic stereogram approach may 
be desirable when presenting 3-D images. In general, a 
stereogram consists of a series of two-demens i onal object 
views differing in horizontal point-of -view . These views are 
presented to the participant in the correct horizontal 
location, resulting in the depth cues of stereopsis and 
(horizontal ) mot ion parallax. The 2-D perspective views are 
generally imaged at a particular depth position, and are 
multiplexed by horizontal angle of view. A given holo-line in 
this case contains a holographic pattern that diffracts light 
to each of the horizontal locations on the image plane. The 
intensity for a particular horizontal viewing angle should be 
the image intensity fOor the correct perspective view. This 
is accomplished by making the amplitude of the fringe 
contribution a step-wise x-funtion of the intensity of each 
image point from each of the views. To facilitate rapid 
computation of stereogram-type computer graphic holograms, the 
precomputed tables can be indexed by image x-position and 
view-angle (rather than by x position and z position). 
Summation is performed as each of the perspective views of 
each segment 26a is read into each computer 9al-9nth based on 
the viewpoint of . the participant. The participants viewpoint 
is sensed by a any previously mentioned position sensing 
system, such as a LADAR 97., that comprises the interactive 
input system 1£. Furthermore, the chartges in x can be indexed 
by look up tables of the computer 9. Alternatively, a simple 
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drawing program written by MIT has been written in which the 
user can move a 3-D cursor to draw a 3-D image that can be 
manipulated. Stereogram computer graphic holograms, such as 
those computed and displayed on the MIT real-time holographic 
display system, and which produce realistic images, may be 
computed utilizing sophisticated lighting and shading models 
and exhibiting occlusion and specular reflections in system 1. 

Each computer 9al-9f 9 may comprise the following system and 
methods demonstrated by the MIT, Media Laoratory. The methods 
of bipolar intensity summation and precomputed elemental 
fringe patterns are used in the hologram computation for 
holographic real-time display. A type of computer 9al-9f 9 for 
holographic processing 220 is generally of a type known as the 
Connection Machine Model 2 (CM2) manufactured by Thinking 
Machines, Inc. Cambridge, MA. Each computer 9al-9f 9 employs 
a data-parallel approach in order to perform real-time 
computer generated hologram computation. This means that each 
x location on the holgram is assigned to one of 32k virtual 
processors. (The 16k physical processors are internally 
programmed to imitate 32k "vrtual" processors.) A Sun 4 
workstation is used a front-end for the CM 2, and the parallel 
data programming language C Paris is used to implement 
holographic computation. In this manner, each image segment 
26b 1 -26a 9 may be computed for each associated holographic 
dispaly unit 34al-34f 9 . 

Once each computer 9al-9af has rendered holographic signals 
for output the signals are transmitted from each 
supercomputer. The signals represent the optical fringe 
patterns that would be present in a 3-D hologram composed of 
192 horizontal lines. Each line is composed of 16 other lines 
and has 32,000 picture elements. To simplifiy the computing 
task, som information is omitted from the hologram. The light 
signals are converted into a radio-frequency signal, 
amplified, and sent to three transducers attached to a 
tellurium-dioxide acousto-opt ical crystal. Here the signals 
are converted into sound waves that travel through the crystal 
at about twice the speed of sound, altering the index of 
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refraction as they change pitch and advance. A laser beam 
passing throught the crystal is defracted just as it would be 
passing through a hologram. A spinning mirror — 18 facets on 
the edge of a thick brass polygon--"f reezes" this holographic 
information. The mirror spins in th opposite direction and at 
the same speed as the sound waves. When a ciruit counts 16 
lines, it sends a signal to a vertical scanner, an 
electrmechnaically driven mirror that moves down one step for 
the next scan line. A lens focused on the spinning polygon 
magnifies the 192 scan lines into a visible image. Simple 
images, such as a wire-frame cube, can be updated by the 
computer in about 0.3 second, so animation is possible, 
showing the 3-D cube spinning against a black background. A 
series of animated images that is precomputed, may be stored 
on magnetic discs, then rapidly downloaded for smooth, 
flicker-free animation. The front viewing face of each 3-D 
display unit 90al-90f9 is faced inward toward the participant 
such that a panoramic scene of speherical coverage is 
presented to the participant. 

If the images are accurately computed and registered, the 
resulting image looks like a solid three-dimensional subject. 
Such a composite or synthetic hologram is termed a 
"holographic stereogram." It mimics the visual properties of 
a true hologram even though it lacks the information content 
and inter ferometr ic accuracy of a true hologram. 

It is forseen that other types and improved holographic 
processors and larger holographic display units similar to 
that described above may be incorporated in the present system 
1. It is further forseen within the scope of the system 1 
that similar and other holographic processors and diplay units 
may operate on the same basic image based virtual model for 
holographic image generation, and use the basic assembly ± 
arrangement for holographic display. It is also forseen that 
projection hologrphic display units may be situated about the 
viewing space 58. to project 3-D images into the viewing space 
to add increased realism to the images Viewed by the 
participant. 
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4) TELECOMMUNICATIONS EMBODIMENT 

FIGS. 2_0 and 2_1 illustrates an embodiment in which system 1 
is used in a telecommunications application. A virtual 
reality/telepresence teleconf erencirg system 2_0 includes 
computer 9, interactive input system 10, source 2 information 
derived from an input means consisting of a panoramic camera 
system 6_, 3-D panoramic digitizer system X, and 3-D panoramic 
audio system B, a head mounted display assembly 11_ and/or 
large display assembly 12_ ; and telecommunications peripheral 
apparatus 92-96 to allow interconnection of two or more 
terminals to engage in teleconferencing via digital data 
networks 9_8. The digital data telecommunications network may 
be part of a local area network (LAN) or wide area network 
(WAN) known to those in the telecommunications industry. 
Additionally, it is also known to those in the 

telecommunication industry that any suitable telephone, cable, 
satellite, or wireless network 98 compatable with the present 
invention may select, switch, transmit, and route data of 
system 1 between remote locations. Typically a frame grabber 
interfaces each camera of input system 6_ with its computer 9_- 
Signals output from the computer 9. are encoded and compressed 
before being input to a telephone line via a 

modulator/demodulator 9_4. A decoder 9_6 is conntected between 
the modulator/demodulator 94. and the computer for decoding 
compressed video signals received by the modulator/demodulator 
means from the telephone line 99 so that the signals may be 
displayed on a video display conected to the computer. 

The telecommunications system 20. is configured to send and 
switch from video to high-resolution graphics, or voice modes 
without breaking connection. For example the participant may 
choose to transmit imagery data and/or shape data, or can 
update a virtual environment by simply passing data 
representing position coordinates to update the position of 
actual live 13, or prerecorded modeled l± beings, objects, or 
scenery modeled or stored at remote locations. Alternatively, 
a plurality of computer workstations 9. and telecommunications 
apparatus 92-96 may operate to in parrallel to transmit data 
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over a plurality of telephone lines to facilitate the 
transmission of large volumes of information. For instance, 
high bandwidth imagery of each side of the large display 
assembly 12. may be 

transmitted over a separate telephone line 9_9 to each 
corresponding display unit 7Qa-70f . 

The telecommunications system can be added as an internal 
peripheral to a computer, or may be added as a stand alone 
device which conntects to the computer 9. via a serial or 
parallel interface. Additionally, the telecommunications 
system may be used in a one, two, or many-way ("broadcast") 
mode. U.S. Pat. 5,062,136 by Gattis et al. is generally of 
the type incorporated and compatable with the present system 
1 . 

5) VEHICLE EMBODIMENTS 

It is foreseen that display system 1_ may be used with 
various kinds of vehicles. In FIGS. 22. and 23_ input means 2 
comprises a panoramic camera system 6a-6f , panoramic 
laser-radar (LADAR) system 7a-7f . and a panoramic audio system 
8a-8f . Each LADAR 7a-7f of the system includes a registered 
visible video channel. The LADAR system searches, acquires, 
detects, and tracks a target subject 13_ in space. The LADAR 
initially searches a wide field of view 223 . The LADAR 
includes focusable optics that subsequently may focus. on a 
subject in a narrow field of view 224 . Once the subject 12 is 
resolved each LADAR video processor 15a-15f associated with 

each LADAR system 3-a — 74- switches from a search mode to 
determining the subject 13. orientation and position. A vision 
processor 15a-15f (e.g. a SGI Computer Workstation or 
Macintosh PC) of each LADAR system includes an object 
recognition capability which correlates, identifies, and 
registers subjects detected by each LADAR's laser ranging 
system. Imagery from each LADAR is fused with the camera 
imagery 6a-6f by a computer 15a-15f . The fused data from each 
computer 15a-15f is transmitted to an associated display unit 
70a-70f of assembly means 4_. The scenes are displayed about 
the viewer /operator 2±. Each input source, computer fusion 
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processor, and display unit operates on an image with a 90 
degree field of view. All systems may be synchronized by the 
master clock integral to any one of the computers 15a-15f . 
This is typically done by conventional software program t~ 
synchronize the signals of the machines 15a-15f and the use of 
common wiring network topologies such as an Ethernet, Token 
Ring, or Arcnet to interconnect machines 15a-15f . 

FIG. 2_3 illustrates the arrangment as configured on a module 
of a space station. It should be understood that various 
system 1 arrangements can be placed on any suitable vehicle 
platform. The participant operates interactive control 
devices 101 to pilot the host vehicle 102 in response to the 
audio and visual scenery displayed around the participant. 
Additionally, the system 15 may include object recognition 
processors. A LADAR 7a_ and camera 6a of a type compatable for 
incorporation with the present invention is manufactured by 
Autonomous Systems Inc. of Orlando, FL . 

Similarly it is forseen that the system could be used to 
remotely pilot a vehicle, such as described in FIGS . 24 and 
2J5. In FIG. 2J3 a 3-D representation of the scene is recorded 
by a sensor array 36a-36f such as that described in FIGS, 6 
and 7. The sensor array housing 40. is incorporated into the 
outer skin of a remotely piloted or teleoperated vehicle 108 . 

In FIG. 25 the vehicle 106 incorporates a video data 
compression system 226. to transmit visual informat ion sensed 
about the vehicle over the air to operate a control station 
225. An over the air radio frequency digital communications 
system 226 transmits 1000 to one compressed full color signals 
at a 60 hertz data transmission rate. Imagery data from a 
panoramic camera system 6. is transmitted to an output 
communications buffer. The frame buffer reads out the digital 
data stream to the radio frequency tranceiver 109b for 
transmission to the transceiver 109a . Tranceiver 109a is 
located at the control station 225 . The transceiver 109a 
recieves the over the air signal and reads the signal into the 
input communications buffer. The input;, communications buffer 
reads out a digital data stream to a data signal decompression 
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for data expander) device. The decompressed signal is read to 
signal processing unit 3 for processing and distribution. The 
processed image is transmitted from processor 3 to display 
units 11 or 12. Shape data and audio data may also be 
transmitted over the digital over the air data link. A 
teleoperated vehicle 22£ data transmission and control system 
compatable with the present system 1 is manufactured by 
Transition Research Corporation of Warren, MI. The panoramic 
camera system 6_ like that in FIGS. 2^7 replaces the camera 
system of the Transition Research Corp. camera arrangement. A 
single or plurality of channels may comprise the system 2_1. 

The participant 2± of the control station 225 interacts with 
the real world remote environment by viewing the displayed 
scene and operating devices of the interactive input system 
10. The control signals are transmitted from the system 10, 
from tranceiver 109a to tranceiver 109b to vehicle control 
systems 112. The control system includes data processing 
means that operates on the transmitted signal to actuate 
control surfaces and motors 113, and manipulators onboard the 
teleoperated vehicle. In this manner the participant remotely 
controls the teleoperated vehicle 108 . 

It is further forseen that the sensor array 36, may be 
mounted onboard unpiloted vehicles such as robots. The 
sensors of each sensor array would be faced outward from the 
robot. The sensors of each sensor array would be in. 
communicating relationship with the robots computer processing 
means. Sensed data would be fused and operated upon by the 
robot to assist the robot in negotiating and interacting 
within it's environment. 

DISPLAY ASSEMBLY MEANS 
1) HEAD MOUNTED DISPLAY ASSEMBLY 

The processing means 1£ of computer 9 generates signals 8£ 
transmitted to the HMD H via conductor lines and these are 
converted into two images on respective* high-resolution, 
miniature display units 70a-70b housed within the HMD 
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assembly. The display units are mounted on opposite sides of 
the HMD assembly in direct communication with the respective 
left and right eyes of the viewer wearing the HMD assembly. 
HMD assemblies of the type 

generally compatable with the present invention are 
manufactured by VPL Research Inc. of Redwood City, CA as the 
EyePhone TM HRX System; by LEEP Technologies Inc., Waltham, MA 
as the Cyberspace 2; by Sony Corporation, Inc. of Japan as 
VisorTron TM; and that sighted in U.S Pat. 5,034,809 by Katoh 
and assigned to Palca, Inc, Tokyo, Japan. 
2) LARGE DISPLAY ASSEMBLY 

Alternatively, a large display assembly V2 is provided to 
receive and display the virtual model to a viewer. The large 
display assembly is configured in a polyhedral arrangement. 
The display assembly comprises a structural assembly that 
encloses the participants head, upper body, or entire body. 
The assembly is designed to facilitate a single or plurality 
of viewers. The floor 101 and its' associated display units 
beneath, to the sides and over the viewer are integrated so 
the participant is presented with a substantially continuous 
scene for viewing. The structural framework, supports, and 
associated fasteners 107 are integrated with the display 
assembly such that they are hidden from the participant and 
hold the assembly together. Display systems and optical 
enlargement means mounted on spring-hinged doors, latches, or 
rollers, allow the entry and exit assemblies 106 to move back 

and forth in an open and closed position. The floor on 
which the viewer is situated is preferably of a rigid 
transparent material through which the viewer sees the viewing 
side of the display systems or optical enlarging assemblies. 
Alternatively, the viewing side of the display systems or 
optical enlarging assemblies is constructed of materials that 
support the participant. The material on which the viewer is 
situated is preferably formed of a transparent rigid glass, 
plastic, or glass-plastic laminate. 

The viewing surface 81, of the display units may be curved or 
flat and face inward toward the center of the viewing space. 
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The diplay units TO typically comprise image projection units 
and associated rear projection screens, cathod ray tube 
display units, or flat panel displays. Single display units 
70a ' 70 ^ ma Y comprise a side of the viewing space, or a 
plurality of display units 70al-70f 9 may make up a side of the 
viewing space 58 . 

As shown in FIG. 19. stereographic display unit 3_2, 
autostereoscopic display units 32, or holographic display 
units 33., and associated screens, audio components, and entry 
and exit ways may be supported in the similar manner as 
conventional display units and screens as described in U.S. 
Pat. 5,130,794 by the present inventor. 

As shown in FIG. 22 the entire display assembly 11 or 12 may 
be located on any suitable vehicle. 

It is to be understood that while certain forms of the 
present invention have been illustrated and described herein, 
it is not to be limited to the specific forms or arrangement 
of parts described and shown. 

CLAIMS 

What is claimed and disired to be secured by Letters Patent 
is as follows: 



1. A display system for virtual interaction with said 
recorded images comprising: 

(a) input means including: 

(1) a plurality of positionable sensor means of 
mutually angular relation to enable substantially 
continuous coverage by said sensor means of a given 
three-dimensional subject ; 

(2) sensor recorder means communicating with said 
sensor means and operative to store and generate 
said sensor signals representing a said subject; and 

(b) signal processing means communicating with said 
sensitive means and said recorder ifieans, receiving said 
image signals from at least one of said recorder means, 
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and operable zo texture map virtual images represented 
by said image signals onto a three-dimensional form; 

(c) a panoramic audio-visual display assembly means 
communicating with said signal processing means and 
enabling the display to a participant of the texture 
mapped virtual images; and 

(d) participant control means communicating with said 
image processing means and enabling interactive 
manipulation of said texture mapped virtual images. 

2. A system according to claim 1_ wherein: 

(a) the plurality of sensor means are positioned 

in an inwardly facing mutually angular relation to 
enable substantially continuous coverage by said sensor 
means of a given, sub ject . 

3. * A system according to claim 1_ wherein: 

(a) the plurality of sensor means are positioned 
on a housing means in outwardly facing, mutually 
angular relation to enable substantially spherical 
coverage by said sensor means. 
£. A system according to claim 1 where said input means 
comprises: 

(a) a plurality of positionable objective lens means 

of mutually angular relation to enable substantially 
continuous coverage by said lens means of a given 
subject; 

(b) light sensitive means optically communicating with 
said lens means and receiving respective optical 
images therefrom and generating image signals 
representing said optical images; and 

(c) image recorder means communicating with said light 
sensitive means and operative to store said image 
signals. 

5. A system according to claim 1 where said input means 
comprises: 

(a) a plurality of microphones positioned in mutally 

angular relation to enable continuous coverage by said 
microphone means; and 
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(b) audio recorder means communicating with said 

microphones and operative to store and generate audio 
signals. 

&_. A system according to claim 1 where said input means 
includes: 

(a) a plurality of three-dimensional digitizing sensor 
means positioned in mutually angular relation to enable 
continuous coverage by said digitizer means; and 

(b) three-dimensional digitizer recording means 
communicating with said digitizing sensor to store and 
generate signals representing a three-dimensional 
representation of said subject. 

1_> A method for recording a panoramic computer generated 
virtual environment comprising the following actions: 

(a) digitizing the shape of all beings, objects, and 
scenes to be represented as part of the virtual 
environment; 

(b) imaging all beings, objects, and scenes to be 
represented as part of the virtual environment; and 

(c) recording audio representations eminating from all 
beings, objects, and scenes. 

8_. A system according to claim 1 wherein signal processing 
means includes: 

(a) an apparatus for adapting video and digital signals to 
a computer in a telecommunications system to allow a 
plurality of computer terminals to engage in 
teleconferencing via a digital data network. 
A method for rendering a three-dimensional computer 
generated model comprising the following steps: 

(a) recording substantially all aspects of a subjects 
shape, visual, and audio signature by recording 
segments representing each aspect of a subjects shape, 
visual, and audio signature; 

(b) constructing a three-dimensional model of each segment 
by fusing corresponding shape and visual signatures 
together and assigning cor respond ifig audio signatures 
to each fused shape and visual signature segment; 
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I c ) constructing a panoramic three-dimensional model by 
fusing together said corresponding fused shape and 
visual signature segments, and assigning said 
corresponding audio signatures to fused shape and 
visual signature segments; 

(d) processing said panoramic three-dimensional model 
representing the shape, imagery, and audio signature of 
said subject by use of a digital computer such that 
visual and audio signals are output to a display means; 

(e) communicating said panoramic audio and visual model to 
the participant by an audio system and display ; units, 
comprising a panoramic audio-visual display assembly 
means; and 

(f) participant manipulation of participant interactive 
input devices communicating with said computer 
processing means to effect audio and visual model. 

10.. A non-contact sensor array for recording a 
three-dimensional signature comprising : 

(a) a visual input sensor which records a video signal at 
least representing a field of coverage similiar to that 
of an associated audio sensor and shape sensor, and 
conductor and interface means for transmitting said 
visual signal to an associated processing or display 
means; 

(b) an audio input sensor which records an aural singal at 
least representing a field of coverage similiar to that 
of said associated visual and shape sensor, and 
conductor and interface means for transmitting said 
audio signal to an associated audio processor or 
speaker system; 

(c) a shape input sensor which records a shape signature 
at least representing a field of coverage simialr to 
that of said visual and audio sensor, and conductor and 
interface means for transmitting said signature to an 
associated processing or display means; and 

(d) housing means to hold said visuaj., audio, and shape 
sensors in place and in a mutually communicating 



64 



WO 94/16406 



PCT/US94/00289 



relationship with said subject, and in a communicating 
relationship with said processing or presentation 
means. 

11 . A non-contact sensor array according to claim 8_ 
including: 

(a) a plurality of arrays oriented in an outwardly facing 
mutually angular relation to enable substantially 
spherical coverage of a three dimensional subject. 

12 . A non-contact sensor array according to claim 8_ 
including: 

(a) a plurality of arrays oriented in an inwardly facing 
mutually angular relationship to enable substantially 
continuous coverage of a three dimensional subject. 

13 . A system according to claim 1 wherein: 

(a) a digital computer including a pixel programmable 
display generator; 

(b) said video display generator being operatively 
connected to said display assembly means; and 

(c) graphics input means operatively connected to said 
computer to be operated by a viewer to cause the 
generation, alteration, and display of images by 
display assembly means. 

14 . A system according to claim .1 wherein signal 

processing means includes: 
(a) an expert system and complementary data base. 
15 . A system according to claim 1_ wherein signal processing 
means includes: 

(a) an expert system consisting of retrieval rules, which 
each associate one of several attributes to an object 
in accordance with the values of inputs; 

(b) analysis rules, which sellectively associate an 
attribute with an object, and which are somewhat 
anogous to the natural-language inference rules which 
would be used in communications between domain experts; 
and 

(c) action rules, which select ively 'carry out the output 
and control actuation options, based on the attributes 
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associated with objects by the other rules. 
16 ■ A system according to claim I wherein signal processing 
means includes: 

(a) a database vithrn the host computer that assigns 
attributes to subjects modeled in the host computer, 
where the attributes are prompted by actions of the 
viewer operating at least one interactive input device. 
17.. A system according to claim 1 wherein signal processing 
means comprises: 

(a) a plurality of host computers which each process a 

segment of the panoramic subject for display. 
18.. A system according to claim 1 wherein signal processing 
means and display unit means includes: 

(a) processing means to generate two independent views 
taken from viewpoints a little distance apart of a 
portion of said images mapped onto said 
three-dimensional form and communicate those views as 
stereoscopic video signals; and 

(b) a plurality of display units of said assembly means 
operable to display said signals on each respective 
display unit as a stereoscopic image to said 
participant . 

19.. A system according to claim 1 wherein signal 

processing means and audio-visual display assembly 
means includes: 

(a) an autoster eoscopic audio-visual display system in which 
the participant requires no eye glasses. 

20.. A panoramic autostereoscopic audio-visual display 
assembly comprising: 

(a) structural support means for securing said 

autostereoscopic display units about the viewer, 
supporting the planar floor on which the viewer is 
situated, and supporting entry and exit portion of said 
assembly; 

(bj said plurality of autoster oscopic dislplay units 
supported by said structural support means such that 
said display units face inward to the viewer, said 
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display units operable to display autoster eoscopic 
images, said display units arranged to form sides 
tangent to one another about a viewer such that a 
viewer views a respective portion of said composite 
image in substantially any viewable direction 
surrounding the viewer; 

(c) said rigid planar floor positioned close to the lower 
side, of said assembly, said planar floor adaptive to 
viewing said autoster eoscopic images eminating from 
said display units by said viewer, and said planar 
floor supporting said viewer; and 

(d) said entry and exit portion of said assembly 
positioned latterly of said viewer. 

21 . A system according to claim 1 wherein signal processing 
means and audio-visual display assembly means include: 
(a) a holographic display system. 

2_2. A panoramic holographic audio-visual display assembly 
comprising : 

(a) structural support means for securing holographic 
display units about the viewer, supporting the planar 
floor on which the viewer is situated, and supporting 
entry and exit portion of said assembly; 

(b) said plurality of hologrgaphic dislplay units 
supported by said structural support means such that 
said display units face inward to the viewer, said 
display units operable to display holographic images, 
said display units arranged to form sides tangent to 
one another about a viewer such that a viewer views a 
respective portion of said composite image in 
substantially any viewable direction surrounding the 
viewer; 

(c) said rigid planar floor positioned close to the lower 
side of said assembly, said planar floor adaptive to. 
viewing said holographic images eminating from said 
display units by said viewer, and planar floor 
supporting said viewer; and * 

(d) said entry and exit portion of said assembly 
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positioned latterly of said viewer. 
2Ji< .A system according to claim 1 wherein signal processing 
means includes: 

(a) a voice recognition system. 

24.. A system according to claim 1 wherein signal processing 
means includes: 

(a) a position sensing system. 

25. A system according to claim 1 wherein signal processing 
means includes: 

(a) image segment circuit means for partitioning an image 
output by said computer processing means. 

2£. A system according to claim 1 wherein the audio-visual 
display assembly includes: 

(a) a plurality of flat panel display systems. 

2J7. A system according to claim 1 wherein the audio-visual 
display assembly includes: 

(a) a plurality of cathode ray tubes. 

28.* A system according to claim 1 wherein the audio-visual 
display assembly includes : 

(a) a plurality of video projection display units and 
associated projection screen. 

29.. A system according to claim 1 wherein the panoramic 
audio-visual display assembly includes: 

(a) a head mounted display assembly. 

3_0. A system according to claim 1 wherein the panoramic 
audio-visual display assembly comprises: 

(a) structural support means for securing display units 
about the viewer, supporting the planar floor on which 
the viewer is situated, and supporting entry and exit 
portion of said assembly; 

(b) said plurality of dislplay units supported by said 
structural support means such that said display units 
face inward to the viewer, said display units operable 
to display images; 

(c) said display units arranged to form sides tangent to 
one another about a viewer such th^t a viewer views a 
respective portion of said composite image in 
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substantially any viewable direction surrounding the 
viewer; 

(d) said rigid planar floor positioned close to the lower 
side of said assemMy, said planar floor adaptive to 
viewing said images eminatihg from said display units 
by said viewer, and planar floor supporting said 
viewer; and 

(e) said entry and exit portion of said assembly 
positioned latterly of said viewer. 

li- A system according to claim 1 where said audio-visual 
display assembly comprises: 

(a) a head mounted display assembly receiving a processed 
video signal which has been processed by said 
processing means for display on said head mounted 
display assembly; and 

(b) a second audio-visual display assembly including a 
plurality of display units receiving respective 
processed video signals representing image segments of 
said composite image from said segment circuit means; 
and 

(c) each said assembly means being arranged such that a 
viewer views a respective portion of said scene of 
substantially spherical coverage in any viewable 
direction. 

12. A system according to claim 1 wherein participant 
control means includes: 

(a) a non-contact participant position sensor system. 

13. A system according to claim 1 wherein the 
signal processing means includes: 

(a) a target recognition system. 

34.. A system according to claim 1 wherein the signal 
processing means includes: 

(a) a synthetically generated three-dimensional auditory 
display system. 

35. A system according to claim 1 wherein the signal 
processing means includes: 

(a) a video teleconferencing system. 
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3C. A system according to claim 10 wherein said housinq 
means comprises: 

(a) the exterior surface of. a host vehicle. 
" ' 37.. A system according to claim 10 wherein said housing 
means comprises: 

(a) the exterior surface of a teleoperated vehicle. 

38.- A system according to claim 10 wherein said housing 
means comprises: 

(a) the exterior surface of a robotic vehicle. 

3£. A system according to claim 1 wherein the system 
includes: 

(a) a conventional stereo over-the-air audio transmitter 
transmits a stereo audio signal to a conventional 
stereo over-the-air receiver of conventional stereo 
headphones worn by a participant. 
37.. A system according to claim 1 wherein the third 
processing means comprising the host computer for manipulation 
of the world model includes: 

(a) associated software for assigning action to subjects 
in computer generated world model based upon actions by 
another subject in the computer generated world model. 
3JB- A system according to claim 1 wherein the forth 
processing means of the system includes: 

(a) a processing means for sampling out a plurality of 

independent views of the computer generated world model 
and transmitting said views as separate video signals 
to a next device for processing or display. 
39.. A system according to claim 1 wherein the signal 
processing means includes: 

(a) a television production system. 

4JD. A system according to claim 1 wherein the signal 
processing means comprises: 

(a) a digital computer system. 

11. A system according to claim 1 wherein the signal 
processing means includes: 

(a) a knowledge based artificial intelligence system; and 

(b) a peripherial device in communicating relationship 
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with said knowledge based system such that the 
peripheric?! device operates and performs functions in 
the real world based on changes and actions of subjects 
in the virtual world; the subjects perceptions being 
formed by a subject beings knowledge based systems 
perception of the real world sensor data modelded in 
the- virtual model . 

11. A system according to claim 1 wherein the signal 
processing means includes: 

(a) a participant mechanical feedback device, actuated by 
particpant actions and actions by subjects programmed 
into the virtual model. 

45.. A system according to claim 1 including: 

(a) a motion simulation system. 

11- A method for simultaneously generating a three- 
dimensional computer generated model and tracking the position 
and orientation of a real world subject including: 

(a) positioning shape, image, and acoustical sensors to 
record respective sensor representation of all sides of 
a given subject; and 

(b) processing same sensor data of said subject to first 
derive position and orientation of the subject and 
secondly to render three-dimensional computer generated 
model of said same subject. 
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